52021SC0236

-the development of technical capacity at Member State level, understood as a reduction in the requests for advisory services between the two programming periods.

On the ESIF mandate, JASPERS’ portfolio of assignments for technical advisory in 2014-2018 included 414 assignments for major projects. Of these assignments, 178 were completed by the end of 2018 and an additional 48 were completed in 2019. Of the 178 major projects covered by the completed assignments, 158 were adopted by the EC by October 2019. The adopted projects assisted by JASPERS involved a total EU funding of EUR 23.9 billion (ERDF and Cohesion Fund combined), representing 65% of EU resources mobilised for major projects adopted by the EC during the reference period of the evaluation.

Compared with the first 5 years of 2007-2013, in 2014-2018 JASPERS completed fewer advisory assignments for major projects in the period under review (178 compared to 343 in the first 5 years of the previous programming period). Despite this smaller number of assignments, JASPERS helped mobilise 22% more EU resources in 2014-2018 than in 2007-2013. The variations in the number of assignments between Member States and sectors is detailed on page 35 of Annex 4 of the SWD.

On the rates of interruption, there has been a significant reduction (from 82% to 46%) in interruptions of major projects assisted by JASPERS advisory services during approval procedures 39 compared to the previous programming period.

Comparing projects assisted and not assisted by JASPERS advisory services, the rate of interruption for assisted projects is 48% as against 43% for the non-assisted ones. However, the higher rate of interruption for projects assisted by JASPERS was not attributable to any failings of JASPERS advisory services, but rather to: (i) the features of the projects concerned (such as a higher level of complexity, which underpinned the need to seek external advice); and (ii) country-specific factors (such as the level of technical administrative capacity). For example, projects assisted by JASPERS advisory tended to have lower rates of interruption than non-assisted projects for all sectors, except for the roads sector. For the water and waste water sector, all projects submitted had been assisted by JASPERS advisory, and therefore a comparison was not possible. Within the group of assisted projects across sectors, the rate of interruption tended to be higher for energy and solid waste projects and for water and wastewater projects.

When comparing rates of interruption for assisted and non-assisted projects at country level, it was found that the projects assisted by JASPERS advisory tended to have zero or lower rates of interruption in 4 countries (Austria, Czechia, Portugal, Slovakia), similar rates of interruption for 3 countries (Spain, Italy, Poland), and a higher rate for only 1 country (Greece). Table A4.6 of Annex 4 to the SWD has more detail on this.

An analysis was made of documents produced when providing technical advisory services. This analysis showed that JASPERS’ experts signalled that changes were needed in the documentation in over 57% of the projects that later received ‘observations’ from the EC 40 . This suggests that, even though the JASPERS’ experts included significant issues and comments in their reports, in some cases the issues were not resolved by the Member State before submitting the project to the EC. There are many possible reasons why the comments by the JASPERS experts were not fully resolved by the Member States. One possible explanation is that the experts’ message on issues requiring further attention might not have been sufficiently strong for the Member State to understand that changes were needed. Another possible reason is that there may have been time constraints on the side of the Member States for submitting the project. Yet another possible reason is that political decisions could have been taken not to address the issues signalled by JASPERS at that point. There were also cases where major projects were interrupted and received observations from the EC, even though JASPERS advisory provided a positive assessment of the project, with no suggestions for improvement. This is the case, for instance, as regards the assessment of environmental issues, indicators, and project descriptions, for approximately 90% of the major projects interrupted for issues on these topics.

By comparison, and as noted above, in the 2007-2013 programming period, the overall rate of interruption of major projects assisted by JASPERS advisory was much higher (82% of all projects submitted), with similar rates of interruption for both assisted and non-assisted projects. Because the extent of the reduction in interruptions between these two periods is roughly the same for both assisted and non-assisted projects, this improvement can be seen as a consequence of the changes in the administrative set-up for assessing and approving projects instituted by the EC for 2014-2020, the latter implicitly integrating the involvement of JASPERS services for the review of all major projects.

Finally, on interruptions, an analysis was carried out to assess whether the probability that a project was not interrupted was linked to the stage in which JASPERS’ advisory support was requested for project preparation. This analysis took explicit account of other factors such as project size, sector and country-specific factors. The analysis covered 95% of all new major projects approved by the EC under the submission procedure (entailing the PSA review) up until October 2019 41 . The analysis indicates that major projects were more likely not to be interrupted during the approval/adoption process if they had been supported by JASPERS advisory during the preparation phase. This effect is stronger and more statistically significant when JASPERS advisory provided assistance from the very beginning of the preparation of the project – or at least from half way through its preparation – rather than at a later stage.

In addition, JASPERS’ advisory support was positively correlated to a shorter duration of the preparation of the project relative to the overall project lifecycle, but only when the support was requested at the very start of the preparation stage. At the same time, the evaluation found that later involvement of JASPERS advisory in project preparation might have been correlated with a longer duration of project preparation.

For non-major projects, the effectiveness of JASPERS’ advisory cannot be established, because data on the progress of projects to approval and implementation is not available. However, preliminary evidence for a third of completed assignments indicates an approval rate no higher than 60%.

The detailed analysis underpinning the above findings can be consulted in Chapter 5.2 (Technical assistance for major projects) of Annex 4 to the SWD.

On the review functions, the two strands of work were different in their scope and purpose as set out in the two bullet points below.

·Under Article 102(1) (notification procedure, entailing the IQR review), the full documentation of the major project was assessed by JASPERS before submission to the EC. The EC then received the IQR report issued by JASPERS and a notification template filled in by the Member State.

·Under Article 102(2) (submission procedure, entailing the PSA review), the full documentation of the major project was submitted to the EC.

The choice of one of the two procedures for assessing the major projects was in the remit of the Member State. An important difference between the two procedures is that the Member States were permitted to make changes to the project documentation during the drafting of the IQR report. This was not the case for the submission procedure entailing the PSA report (where changes were possible only based on an official letter of interruption sent by the EC). Implicitly, depending on whether the Member State took on the advice of JASPERS, the IQR process had the potential to lead to faster processing and a higher approval rate of the major projects, and therefore a faster start to implementation on the ground.

Moreover, the median total duration of the process for assessing and approving major projects notified under Article 102(1) (notification procedure, entailing the IQR review) was 7 months 42 , of which the median duration of the IQR assessment was 5 months. By contrast, the median total duration of the assessment and approval process for major projects submitted under Article 102(2) (submission procedure, entailing the PSA review) was 3 months for projects not interrupted and 6 months for those interrupted.

In this process, JASPERS was highly effective in delivering timely PSA reports. Compared to 2007-2013, there was a significant reduction in the overall time taken by the EC to assess and approve major projects in 2014-2018 43 . This reduction was greater for the submission procedure with PSA review (50%) than for the notification procedure with IQR review (13%). By comparison, in 2007-2013, the average duration of the approval process for a major project was around 9 months, and the minimum duration recorded was 6 months.

Details on the ability of JASPERS IQR services to signal issues related to project quality can be consulted in Chapter 5.2 (Review function) of Annex 4 to the SWD.

For capacity building and horizontal and strategic advice, JASPERS’ portfolio included 220 assignments, 75% of which were for horizontal and strategic advice and 25% for capacity building.

The objective of the capacity-building assignments was: (i) to provide training, workshops and online fora on topics relevant for preparing and implementing projects; and (ii) to help disseminate guides and studies. The training and events organised for capacity building could have been either multi-country or tailored to the specific needs of an individual country. In the period analysed, most assignments for capacity building were multi-country assignments, and the most common topics of training/workshops were climate change, environmental requirements and state aid.

During the 2014-2018 evaluation period, most activities for capacity building were completed and achieved their objective of delivering training to the Member States. However, these activities did not prove sufficient to trigger a visible knock-on effect on the technical capacity of Member States that would allow them to effectively address issues in their preparation of projects such as: (i) cost-benefit analysis; (ii) climate change; and (iii) environmental requirements.

It is helpful to compare the main issues addressed by capacity-building activities completed up until end-2018 on the one hand with the quality gaps identified in project preparation during the advisory work of JASPERS and during the assessment of major projects by the EC on the other. Two main issues were identified in this comparison, both of which are addressed in the paragraphs below.

Firstly, despite the relatively frequent focus of capacity-building activities on cost-benefit analyses, climate change and the environment, shortcomings in these aspects of project applications were still among the most frequent reasons for interruptions issued by the EC in the approval process of major projects. This suggests that the capacity-building activities organised on these topics did not succeed in having a significant effect on the technical capacity of Member States to address these issues themselves without support.

Secondly, the documentation of advisory work and the EC’s assessments of major projects showed that there were still many projects in need of advisory support on topics such as feasibility studies, risk analysis, and sensitivity analysis. Capacity building is not that frequent in these three areas. During a consultation, stakeholders suggested that capacity building is better strengthened with hands-on experience while working directly with JASPERS on the projects, rather than through training and workshops. While, in principle, there might be merit in this argument, there is little evidence that this hands-on experience is reflected in better preparation of projects across sectors and countries. This may be due to high turnover of staff in national authorities’ administrations, a phenomenon also referred to in the ECA report. It may also be because JASPERS often interacted with consultants contracted by Member States to prepare project documentation, and therefore the learning effect for the staff of national authorities was limited. Moreover, the evaluation highlighted that countries’ reliance on JASPERS advisory assistance for project preparation has not significantly decreased compared to the previous period. This suggests there has only been a modest capacity increase in project preparation at national level between the two periods 44 .

A pragmatic approach implemented by JASPERS for technical capacity building was to ‘train the trainers’. This focused training on local trainers who could subsequently disseminate the know-how they had acquired to larger audiences on many occasions within and across countries. This approach was used by JASPERS for some capacity-building assignments over the period, but it had not yet been mainstreamed at the level of the portfolio, within the timeframe of this evaluation.

On horizontal and strategic support for capacity building, these assignments were highly diverse in their specific objectives (including support for national strategies on specific issues, river basin flood management plans or integrated urban development plans). For this reason, it was far more difficult to assess the overall effectiveness of the support. An analysis of 10 sampled capacity-building assignments indicated that JASPERS’ support helped compensate for weaker local technical capacity to guide and supervise tasks on specialised topics of policy relevance. However, this finding cannot be generalised for all 166 training assignments for horizontal and strategic support from the reference period of the evaluation.

The detailed analysis underpinning the findings from above can be consulted in Chapter 5.2 (Capacity building and horizontal and strategic advice) of Annex 4 to the SWD.

On the CEF mandate, most 45 of the projects supported by JASPERS during its first pilot mandate were successful in CEF competitions up to August 2019. Seven of the investment projects assisted by JASPERS at the application stage for CEF competitions had also been assisted by JASPERS technical advisory in their preparation in the 2007-2013 programming period under the ESIF mandate. Because of its pilot nature, a comparative analysis of the effectiveness of JASPERS’ intervention under the CEF mandate compared to a previous existing baseline cannot be undertaken. It is also not possible to conduct a comparative analysis of the effectiveness of the intervention for the ESIF mandate. Nevertheless, the objectives stated in the strategic documents have been followed, with JASPERS providing technical assistance to prepare eligible projects, with the intention of them being submitted in response to CEF calls.

On the IPA mandate, by the end of 2018, only 18% of JASPERS’ assignments resulted in IPA projects being approved for EU funding (17 projects). Overall, by the end of 2018, 38% of the assignments had been completed, 50% were still ongoing, and the rest had been administratively closed. Based on an analysis of a sample of six assignments 46 , there is an indication that the effectiveness of JASPERS’ advisory support was limited in cases where the requests for support arrived late in the project preparation cycle. For these late-arriving projects, it can be concluded that JASPERS advisory could have made a difference to project quality if its support had been requested earlier in the project-preparation process. When JASPERS support was requested in the final stages of preparation (as was the case for three completed assignments by 2018), despite JASPERS’ critical assessment of project quality, no substantial changes were made by the national authorities concerned.

Finally, there is a finding that is linked to the consultation activities and valid for all mandates. This finding is that JASPERS had a significant, and even decisive, effect on project technical specifications and documentation. A large majority of stakeholders (80% of 210 respondents) rated highly JASPERS’ assistance in aligning projects with EU and national policies and in achieving faster project preparation and approval.

The state of play of the JASPERS initiative was assessed by the ECA in a performance audit report published in 2018 47 . The audit covered the first 10 years (2006-2016) of JASPERS’ activities and four Member States: Croatia, Malta, Poland, and Romania 48 . On the effectiveness of JASPERS’ activities, the ECA auditors concluded that these activities contributed to faster project approval and had a positive effect on the quality of projects and their documentation. For the implementation of projects assisted by JASPERS, the ECA auditors found that, while JASPERS’ comprehensive assistance could generally not have an impact on the financial implementation of EU funds, the projects assisted were found to be less frequently affected by legal and regularity errors during the compliance audit exercises carried out by ECA in 2014-2015.

Efficiency

The evaluation of efficiency aims at assessing the relationship between the resources used by an intervention and the changes generated by that intervention (the costs and benefits). Because data on the actual costs incurred by the work on individual assignments is limited (see Section 4), the analysis considers JASPERS efficiency by looking at:

-the administrative closure of assignments;

-the work intensity distributed across assignments 49 ;

-the correlation between human resources, financial resources and the number of assignments.

The analysis below covers only the ESIF mandate, which provided 97% of the total financial resources allocated to JASPERS during the 2014-2020 programming period.

Overall, more than 90% of JASPERS’ assignments were completed in the period 2014-2018, with 9% being administratively closed (i.e. assignments cancelled before completion), usually long time after being initiated. Half of those administratively closed assignments were concentrated in three countries 50 .

A quarter of the cancelled assignments had a duration of more than 50 months, highlighting the long time taken before closure. Moreover, based on the analysis of country plans, it can be concluded that many of the administratively closed assignments had no initial expected date for completion or, if they had such a date, this date was much later than set out in the initial plan. On the reasons for administrative closure, the evaluation identified three main reasons: (i) an agreement between JASPERS and the country concerned to combine the assignment with other ongoing assignments; (ii) closure of the assignment because of a change in context (for example, projects prepared for the ESIF but eventually submitted for CEF); and (iii) a decision by the country concerned to remove the assignment from its plan because it was deemed no longer feasible.

On the intensity of work in relation to the duration of assignments 51 , the evaluation concludes that there is little correlation between these two variables in most cases. This means that assignments that last longer tend to have less intensity of work. For technical advisory support, this can be explained by the fact that the advice during various steps in the preparation and implementation of the project depends on a number of factors (time needed to receive the reply from the Member State, contractual arrangements with external consultants at Member State level, etc.). This finding also holds true for review services. However, it is important to differentiate the two strands, as the PSA services had a high concentration of short assignments (with most being finalised within a month 52 ), while the IQR services had a wider range of durations of assignments (up to 10 months 53 ). PSA assignments remained the most work intensive.

Comparing the resources allocated over the two periods, it is clear that JASPERS has grown significantly, increasing from 56 people in 2007 to 140 professional and support staff in 2018. This reflects adaptation to the demand for JASPERS’ services, which grew and became more diversified in the 2014-2020 period.

Moreover, the change in the structure of the budget indicates a reorientation in JASPERS’ staff policy. In the period 2014-2018, the initiative relied on its own staff to a larger extent, as opposed to the more extensive use of external consultancy services in the previous period. This means that most of JASPERS’ resources were invested in its people, with the share of staff costs increasing from 82% to 91% between the two periods.

Figure 8. Human and financial resources, and assignments over time, ESIF mandate

Source: Inforegio report Analysis and assessment of JASPERS’ Services and Performance 2014-2018, based on data reported in JASPERS’ annual reports 2007-2018.

The left panel in Figure 8 illustrates the number of annual full-time equivalent workers (FTEs) supported in JASPERS and the annual monetary contribution by the EC over the period 2011-2018. These data indicate that, starting in 2014, there was a continuous increase in staff every year over the period, while the EC’s annual contribution increased at the same time. Compared to 2013, the number of JASPERS’ personnel in 2018 was 55% larger and the EC’s annual contribution was 65% larger.

For ongoing assignments, there has been an almost constant, linear increase each year until 2017, while staff numbers continued to increase throughout the period. The increase in the annual number of assignments up until 2017 can be explained by both the rate at which assignments were accepted and their median duration of roughly 2 years. The completion or closure of assignments mostly began to increase in 2016. However, the number of ongoing assignments is an imperfect proxy for the actual work involved. This is because, as explained earlier, the work intensity per assignment varied significantly, and work intensity was not very high for most assignments other than those involving the PSA review 54 . This preliminary evidence therefore indicates a need for further in-depth analysis of the topic to help identify possibilities to optimise the allocation of work and rationalise costs.

In addition, the ECA audit also identified scope for improvement in a number of aspects in the administrative set-up and approach of JASPERS’ activities. These included:

-the role and responsibilities of the main stakeholders, which the ECA believes should be defined better by calibrating more precisely the working arrangements between JASPERS and the Member States for all types of service (including capacity-building activities and horizontal and strategic support);

-the need to implement a better system for monitoring the efficiency of JASPERS’ operations.

The detailed analysis underpinning the above findings can be consulted in Chapter 5.3 of Annex 4 to the SWD.

Coherence

The analysis of the coherence criterion was undertaken from both an internal and external perspective. The internal perspective refers to both: (i) the coherence of the different types of service provided by JASPERS with each other; and (ii) the coherence of JASPERS’ services with EU priorities. The external perspective assesses coordination between JASPERS and other advisory services at EU level providing similar technical assistance to Member States.

The evaluation finds that JASPERS’ services were coherent with EU requirements set by cohesion policy legislation and the EU-level objectives in the area of climate, the environment and competition. The services helped to guide project preparation and supported the assessment of projects in line with these objectives.

Applying for EU co-financing of major projects required the Member States to ensure their projects: (i) were aligned with both cohesion policy objectives and the EU-level objectives of mitigating/adapting to climate change; (ii) accounted for their environmental impacts; and (iii) met all regulatory requirements related to state aid 55 . As a consequence, all JASPERS services were organised so as to align the advisory and review work with these EU priorities.

In the course of the advisory work, depending on the scope of the assignment requested by Member States, JASPERS provided an assessment (or, at a minimum, a description) of the project’s alignment with: (i) EU policy objectives and priorities; and (ii) EU requirements on the environment, climate change, and competition. In this process, JASPERS identified changes needed in project documentation to ensure this alignment, or made suggestions for improvement. The evaluation found that JASPERS provided advice on climate and environmental issues on 54-60% of the projects assisted, while JASPERS addressed state aid issues in about a quarter of these projects.

From an internal coherence perspective, while acting independently during the course of providing the services, JASPERS review services and JASPERS technical advisory were found to be generally coherent with each other in the way they applied EU requirements to project preparation on the one hand and to project assessment on the other. Some scope for further alignment remained, although this was limited to around 10% or less of projects and concerned minimum regulatory requirements for project quality.

The evaluation finds that for a large majority of projects (above 90%), there was full coherence between the two services. Firstly, for projects with no critical issues identified in the PSA report, there were cases for which JASPERS advisory had raised significant issues during project preparation and the Member State addressed these issues before submitting the project to the EC. These issues were therefore no longer present when the project reached the PSA assessment. The two outputs (of the technical advisory and of the review services) were therefore coherent even if they were different. Secondly, on many other projects on which the PSA report did raise critical issues, JASPERS advisory also signalled significant issues during project preparation, but these issues had not been addressed by the Member State by the time the project reached the EC. In these cases, the two outputs were also coherent, as the outstanding critical issues were due to the Member State failing to heed JASPERS’ warnings on the shortcomings of the application.

On external coherence, the evaluation explored the possibility of assessing the coherence and synergies between JASPERS services and other EC initiatives for technical assistance. Under the 2014-2020 multiannual financial framework, there were several centrally managed initiatives for technical assistance, most of them focused on capacity building and project preparation. These initiatives include: the European Investment Advisory Hub (EIAH); European Local Energy Assistance (ELENA); and the European Energy Efficiency Fund (EEEF). JASPERS was one of the largest of these initiatives, with 30% of the total cumulated budget of more than EUR 700 million for centrally managed technical assistance.

The evaluation concludes that, until the end of 2018, the potential for synergies between JASPERS and other technical assistance initiatives were only partly exploited. JASPERS’ Quality Manual mentions the possibility for JASPERS to cooperate in sharing expertise with other technical assistance services. In 2015, following the publication of the European Fund for Strategic Investments (EFSI) Regulation 56 , JASPERS assessed the scope for cooperation with the EIAH and even set up procedures to promote this cooperation. During the reference period of this evaluation, JASPERS cooperated with the EIAH on three assignments to construct and equip regional hospitals in Romania.

The detailed analysis underpinning the above findings can be consulted in Chapter 5.4 of Annex 4 to the SWD.

Relevance

Because of the limitations described in Section 4 of this SWD, the relevance of intervention is assessed mainly in terms of its scope as compared with the previous programming period. The SWD only examines the connection of JASPERS’ work with identified needs in a limited way. More specifically, the needs analysis performed to inform the 2014-2020 working arrangements was based mostly on direct feedback from the Member States, which were also the direct beneficiaries of the technical-assistance services provided by JASPERS.

The main approach for evaluating the relevance criterion was to assess the structure of JASPERS’ portfolio compared with the previous programming period.

For the ESIF mandate, JASPERS remained relevant for project preparation in the 2014-2020 period, and provided more services for more countries than it did in 2007-2013. In addition, JASPERS became relevant for the new function of quality reviewing ESIF major projects.

Similarly, JASPERS was relevant for the objectives set out in the CEF and IPA mandates, which called for the provision of technical assistance for projects that would be eligible for EU funding.

During the programming period, JASPERS’ assistance became available for all Member States and IPA countries and was mandated by three EC services (DG REGIO, DG MOVE, and DG NEAR). Its services included: (i) technical advice for the preparation of major and non-major projects; (ii) capacity-building activities and horizontal and strategic support; and (iii) a new review function for projects supported by cohesion policy. For the first 5 years of the period, JASPERS registered around 1 100 assignments in 26 countries. By comparison, in the previous period 2007-2013, JASPERS provided technical assistance for 14 Member States and IPA countries through just under 800 completed assignments.

Most of JASPERS’ activities for the ESIF mandate were concentrated in 3 main beneficiary countries (Romania, Croatia and Poland) – 2 of which (Romania and Poland) were consulted for the needs assessment to prepare the 2014-2020 period and indicated high needs of support from JASPERS services. When looking at the distribution of the ESIF assignments at country level, 57% of these assignments were carried out in 3 main beneficiary countries (Romania, Croatia, and Poland), 5% were multi-country assignments, and the rest were distributed across 19 other Member States. By comparison, during the previous period 2007-2013, 59% of JASPERS’ activities for this mandate were also concentrated in 3 main beneficiary countries (Poland, Romania, Bulgaria), with the rest of the activities distributed across 11 more Member States.

JASPERS’ new review function was highly relevant, because it assessed the quality of all new major projects submitted or notified to the EC under the CPR legislative framework for the period 2014-2020 involving either Article 102(1) entailing the IQR review or Article 102(2) entailing the PSA review. The evaluation finds that 71% of new major projects were assessed under the submission procedure (with PSA report) during 2014-2018.

On training support to build administrative capacity, the share of training activities for multi-country capacity building on topics of high relevance for all countries increased. Nevertheless, horizontal and strategic support tailored to country-specific needs accounted for most of the support.

Based on JADE data, the evaluation finds that JASPERS carried out 220 assignments for capacity building and horizontal and strategic support up to December 2018, 75% of which were for horizontal and strategic advice. By comparison, in 2007-2013 JASPERS completed close to 140 such assignments, 96% of which were for horizontal and strategic advice. The relative share of assignments for capacity building, therefore, increased from 4% to 25% between the two periods. These assignments can be tailored to the needs of a specific country or can be organised on a multi-country basis for topics of high relevance to several or even all Member States.

For capacity-building training, in contrast to the previous period when almost all assignments were country specific, there was an increase in 2014-2020 in the frequency of multi-country assignments. The most frequent topics for these training activities were climate change, the environment and state aid.

The evaluation finds that the demand for country-specific capacity-building activities was not high during the period examined.

Only 19 country-specific assignments for capacity building had been completed or were ongoing at the end of December 2018 in 10 Member States. More than 60% of these training assignments were devoted to climate-change adaptation and the environment, with the remainder mostly focused on horizontal aspects, planning and implementation.

Finally, the evaluation finds that JASPERS’ support was relevant for meeting cohesion policy objectives, with JASPERS’ general aim of supporting Member States during the project-preparation process being maintained across the last two programming periods. In practical terms, Member States first sent requests for assistance to JASPERS, which then communicated these requests to the EC 57 in order to receive the green light to proceed, which was generally done by tacit agreement. This approach tended to ensure that Member States’ needs for support were addressed, but it did not mean that there was a strategic analysis or prioritisation of requests. As highlighted by ECA auditors, JASPERS has not targeted its activities sufficiently, as it has not decided on clear priority areas for providing assistance to Member States, but has simply responded to demands as they arose.

EU added value

This section includes the main findings from the analysis of EU added value of the JASPERS initiative. First, it should be noted that the nature of the support concerned implicitly embeds the European dimension, as the technical assistance services provided are supported financially by EU institutions (EC and EIB).

The evaluation of EU added value relies primarily on the feedback received from stakeholders responding to an online targeted consultation during summer/autumn 2019, in which stakeholders’ views primarily referred to the general added value of JASPERS. Of the respondents to this consultation, 75% were either project promoters/beneficiaries or managing authorities, while the remaining 25% were consultancies, companies, EC officials, and other types of institutions at national level. As they had first-hand experience with JASPERS services, the project promoters and managing authorities were able to provide informed opinions on the topics related to services provided by JASPERS. This distribution of respondents to the consultation inherently includes a possible bias in assessing involvement with JASPERS services, as project promoters and managing authorities were directly benefiting from the free-of-charge support provided by JASPERS.

The information obtained from stakeholders indicates that JASPERS services were appreciated because of the extensive methodological and sectoral technical expertise that it provided, thanks to the experience JASPERS gained from working with many Member States. According to stakeholders, this expertise cannot easily be replaced as it is not readily available on local markets.

In addition, a large majority of stakeholders considered that, relative to other advisory services, JASPERS’ added value was higher especially in: (i) its support for aligning projects with EU priorities; (ii) the potential for knowledge transfer among Member States; and (iii) its hands-on experience. Different opinions were expressed on aspects such as flexibility in responding to needs and expertise on administrative and procedural issues.

A large majority of managing authorities and project beneficiaries/promoters were of the view that JASPERS’ services will still be needed in the future, even if EC approval of major or strategic projects was no longer required. According to this majority of managing authorities, JASPERS support is necessary for aligning national strategies with EU sectoral policy priorities. Finally, from an operational perspective, stakeholders also stressed that the use of JASPERS services does not require public procurement procedures, and that this was a distinct advantage. However, willingness to pay for these services from their own technical-assistance budgets is rather low 58 .

The synopsis of the stakeholder consultation is included in Annex 2 of the SWD.

Another aspect that is relevant to JASPERS’ EU added value lies in the organisation of its review function. According to the regulatory framework in place, Member States were able to set up individual arrangements for the independent quality review of the project documentation before they submitted their major project to the EC under Article 102(1) of the CPR. The creation of the IQR division in JASPERS, which provides review services available to all project promoters across Member States, reduced the administrative burden for Member States in the area of independent quality reviews.

6.Conclusions

The purpose of this evaluation is to assess: (i) the implementation of the services provided by JASPERS; (ii) the contribution of JASPERS services to the development of high-quality, mature infrastructure projects; and (iii) the contribution of JASPERS to enhancing the technical capacity of managing authorities and beneficiaries in preparing such projects. The evaluation covers all JASPERS activities during the 2014-2018 period across all countries that made use of its advisory services (i.e. Member States and pre-accession countries).

The JASPERS initiative is assessed against the five evaluation criteria decided on by the EC’s better regulation guidelines: effectiveness, efficiency, coherence, relevance and EU added value.

As explained in the previous sections, the effectiveness, efficiency and relevance evaluation criteria were assessed via proxy analysis because of limitations in the definition of objectives and the design of the data monitoring system. The broad definition of JASPERS’ general objective constrained the evaluation, as this definition does not provide an indication of measurable outcomes that could inform the evaluation of JASPERS’ effectiveness and relevance. On the monitoring system, the available data on the exact amount of time worked on assignments started to be recorded only in 2017, while the evaluation covered the period up to the end of 2018. The limited timeframe of available data allows only for having initial reflections on the efficiency criteria.

Overall, the evaluation finds that JASPERS’ activities expanded both in scope and volume during 2014-2018 compared to the previous period. On scope, although the ESIF mandate represented the largest share (close to 90%) of the overall portfolio in numbers of assignments, JASPERS’ services were supported over the period by two more EC services, as DG MOVE and DG NEAR joined DG REGIO in financing the initiative for CEF and IPA projects. JASPERS’ services were available for all Member States and IPA countries. JASPERS’ role was significantly strengthened by the EC through the creation of a new function for the quality review of major new ESIF projects. On volume, in the first 5 years of the 2014-2020 programming period, JASPERS received more requests for services than in the entire period 2007-2013.

Effectiveness

Much of JASPERS’ activities consisted of helping Member States to prepare major projects to obtain EU funding in a timely way. Of the 226 assignments for technical assistance completed up to end-2018, 158 (70%) resulted in major projects being adopted by the EC by October 2019. These accounted for a total financing from the ERDF and Cohesion Fund of EUR 23.9 billion, 65% of all funding going to major projects over the period. This is the same percentage as in the first 5 years of the 2007-2011 period (when more major projects but of a smaller size were financed).

The timing of the involvement of JASPERS’ advisory in project preparation for major projects had a significant effect on the effectiveness of this involvement. Firstly, the evidence indicates that, when compared to non-assisted projects, the projects assisted by JASPERS advisory had better chances of being adopted without interruptions for quality issues. On the other hand, there is no compelling evidence that JASPERS’ support reduced either the number of issues that the EC raised or the duration of the interruptions. Nevertheless, early involvement of JASPERS advisory services in the project cycle was associated with a shorter duration of the core project preparation relative to the overall project lifecycle.

The evaluation found that JASPERS advisory collected limited data on what happened to non-major projects it assisted once the advisory work was completed. This restricted the possibility of assessing the effectiveness of JASPERS’ support.

The evidence suggests that JASPERS was relatively effective in flagging quality issues in the PSA assessments, which were then included by the EC in its letters to Member States setting out the reasons for requesting improvements (i.e. interrupting projects).

While a large majority of activities for capacity building were completed, and achieved their objective of delivering training to the Member States, there is little evidence that they triggered a visible ripple effect in reducing Member State reliance on JASPERS’ support, which should, in principle, be the ultimate objective. The demand for these services increased rather than declined between 2007-2013 and 2014-2020. Issues relating to cost benefit analysis, climate change and the environment, on which training and workshops focused, continued to feature prominently in EC interruption letters. Other issues that also featured extensively in EC interruption letters included feasibility studies and risk and sensitivity analysis, neither of which were much covered in capacity-building activities. However, the evaluation found examples of a pragmatic approach to capacity building, in the form of JASPERS’: (i) ‘training the trainers’ activities at local level as a source of continuing knowledge dissemination; or (ii) coordinating training on specific issues across countries. These activities have the potential to be promoted more widely (as is now recognised by JASPERS management) as part of the strategic planning of capacity-building activities, with the long-term aim of reducing reliance on JASPERS’ assistance for preparing and implementing projects.

Efficiency

When assessing efficiency, it is difficult to set a suitable benchmark against which indicators of the time taken to complete assignments (or the staff effort/work effort involved) can be measured. In other words, although it might be possible to calculate the indicators concerned, it is still difficult to interpret these indicators and determine whether or not they are higher than they should be. This is a problem that is common to assessing the efficiency of cohesion policy interventions in general.

The duration of assignments is a potential indicator of efficiency, but it does not necessarily reflect the amount of time taken, data on which have only been recorded since January 2017. The duration of an assignment is also linked to: (i) the initial quality of documentation; and (ii) the scale of revisions needed to meet minimum quality standards. The analysis of JASPERS’ efficiency was thus limited by the short timeframe of the available data on work per assignment. Before 2017, the monitoring framework at the level of individual assignments provided mostly descriptive information (i.e. without any quantitative data) of the assignment itself.

Work intensity, defined as the days worked relative to the days available over the period of the assignment, amounted to 30% on average per assignment for technical assistance on major projects. In practice, there was little correlation identified between work intensity and the duration of assignments. Preliminary evidence on developments over time indicates a need for further in-depth reflection on possibilities for optimising the allocation of work effort and rationalising costs. However, it should be stressed that the main added value of JASPERS lies in the technical expertise and accumulated experience of its experts, and that ownership of projects and timely availability of project documentation remain the responsibility of project promoters.

Although JASPERS services were generally found to be efficient in completing the assignments initiated during the 2014-2018 period, the evaluation concludes that the planning and management of administratively closed assignments could be improved.

Coherence

During the 2014-2018 period, JASPERS’ services were coherent with: (i) EU requirements set by cohesion policy legislation; (ii) the EU-level objectives for climate and environment; and (iii) regulatory requirements for competition.

The evidence compiled by the evaluation also suggests a high degree of internal coherence between the different independent JASPERS divisions. For over 90% of major projects examined, the technical advisory support provided by JASPERS to Member States in preparing projects was consistent with the conclusions of JASPERS IQR services involved in reviewing the major projects submitted to the EC.

However, for the period examined, limited evidence was found of JASPERS cooperating with other technical-assistance initiatives of the EC, although the possibility of sharing experience was mentioned in its Quality Manual.

Relevance

JASPERS’ advisory services remained relevant in the 2014-2020 period, as reflected by the over 400 assignments requested by Member States up to December 2018. By October 2019, 70% of these assignments had resulted in major projects adopted by the EC – projects which mobilised almost EUR 24 billion worth of investments from the ERDF and Cohesion Fund.

In addition, for the 2014-2020 programming period, JASPERS provided a new kind of support: reviewing the quality of new major projects submitted for EU funding, as required by the regulations, with the aim of ensuring consistency in the assessment and reducing the time taken for it.

JASPERS’ capacity-building activities and provision of horizontal and strategic support continued in the 2014-2020 period. Most of the requests for support were horizontal rather than strategic in nature and occurred just before or just after the beginning of the 2014-2020 period as Member States sought to prepare for this new period.

The evaluation finds that the needs analysis developed for 2014-2020 did not take a sufficiently critical view on the possible bias of the stakeholders involved in the analysis, who were also potential direct beneficiaries of JASPERS’ services.

JASPERS’ activities continued to be demand-driven, assignments being initiated at the request of Member States or pre-accession countries. This left little room for the strategic planning of these activities, and for deploying resources where they could produce the most added-value in terms of the management and implementation of EU funding.

EU added value

The consultation carried out indicates that JASPERS services have been appreciated by stakeholders, because of their technical expertise and EU-wide experience. However, another reason stakeholders appreciated JASPERS was because accessing its support did not entail a public procurement process and it was provided free of charge.

JASPERS’ added value was seen as being especially high in: (i) its support for helping to ensure that projects were aligned with EU priorities; (ii) the potential for knowledge transfer among Member States; and (iii) its hands-on experience in countries.

Moreover, stakeholders considered that JASPERS services will also be needed in the future, not only because of the technical expertise brought by the experts and their EU-wide knowledge, but equally because of their ability to help align national strategies with EU priorities.

To conclude, JASPERS’ EU added value, as indicated by the stakeholders engaged with JASPERS support, primarily lies in the technical expertise brought by its experts. These experts have EU-wide experience acquired while working in many countries and with the EC, and would be difficult to replace.

Overall, in relation to the period 2014-2018, the evaluation reaches the following four conclusions.

-JASPERS technical advisory services contributed to a faster and better preparation of projects leading to a faster mobilisation of EU funds.

-JASPERS’ general objective has been broadly achieved together with all of the specific objectives, with the exception of the transfer of technical knowledge to Member States in view of reducing their reliance on JASPERS advisory services.

-the evaluation aimed to address all the evaluation criteria in accordance with the Better Regulation guidelines, it relied on several proxy analysis due to various limitations; in particular the limitation concerning the availability of data needed for assessing the effectiveness and efficiency of JASPERS services was significant (more structured performance monitoring at the level of assignments only started in 2017, and was further strengthened only after 2018).

-There is scope for a more structured approach to the planning and monitoring of assignments, with: (i) well-defined and targeted objectives; (ii) clear and realistic timelines; and (iii) milestones that enable progress in implementation to be monitored over time.

7.Developments made during the last 2 years of the 2014-2020 programming period and the outlook for 2021-2027

Because the evaluation coincides with the preparation of the new mandate for JASPERS that will cover technical assistance during the 2021-2027 programming period, a short presentation of the developments that took place during 2019-2020 is worthwhile. The following bullet points contain this short presentation.

-To improve effectiveness and internal coherence, JASPERS and the EC continued their efforts to strengthen the coordination and early warning system on issues flagged by the advisory services. To this end, regular meetings were organised between the EC and JASPERS on the one hand, and between JASPERS IQR and JASPERS advisory, on the other. In addition, the IQR process became more formal in 2019-2020, following the example of the submission procedure.

-The demand for country-specific capacity-building activities has increased since 2018, partly because of the need to prepare the 2021-2027 programmes.

-The review-services function will no longer exist for the 2021-2027 programming period, as the new legislation proposed a new assessment system for projects of strategic importance, and this assessment is to be carried out by the Member States.

-The EC is currently developing new working arrangements, which are planned to take the form of trilateral work programmes, negotiated for Member States that plan to use JASPERS regularly. These programmes: (i) will set clear objectives for JASPERS’ support at country level; (ii) will decide on priority areas for the support; and (iii) are expected to lead to better prioritisation of assignments.

-Following the ECA performance audit of JASPERS, the EC drew up an action plan with measures to be implemented starting in 2018. By January 2019, the following measures had been implemented fully or were in the process of being finalised: (i) measures relating to clarification of the roles and responsibilities of the main stakeholders; and (ii) measures relating to the working arrangements of the tripartite cooperation between the EC, JASPERS and the Member States. In line with the audit recommendations, the EC has requested improvements to the monitoring system. A new set of performance indicators and JASPERS has reported regularly on these indicators since late 2018.

-On external coherence, in 2019 it was decided that, for the next mandate, JASPERS should be included in the future InvestEU Advisory Hub. This hub will be a single point of entry for EU-funded technical-assistance initiatives. Currently, the EC is working to implement this decision by adjusting the JASPERS governance system to the context of the Advisory Hub. There was a reorganisation of JASPERS inside the EIB to accommodate these changes.

-Following the CEF pilot experience with JASPERS’ assistance, DG MOVE has decided to support a renewed mandate for JASPERS’ services for CEF projects in the programming period 2021-2027. This renewed mandate has a significantly larger budget (EUR 30 million for the period 2021-2024), and there is therefore increased potential for more extensive assistance to projects in all Member States. DG MOVE will delegate its budget to DG REGIO to promote cross-fertilisation, coordination and synergies between the cohesion policy programmes and the CEF for advisory services of the very same nature on transport projects (including on the trans-European transport network (TEN-T)) in all Member States.

The developments in 2019-2020 have already addressed most of the points for improvement highlighted by this evaluation. The DG REGIO negotiating team for JASPERS’ services for the 2021-2027 programming period has been regularly informed of the progress of the evaluation. The evaluation’s findings and conclusions have been discussed and taken into account by DG REGIO as appropriate. DG REGIO has given particular attention to the more strategic approach of the assistance to be provided by JASPERS (the country-specific trilateral work programmes between the EC, the Member States and JASPERS as explained above).

Two of the main aspects still to be discussed for the JASPERS services in the 2021-2027 period concern: (i) the monitoring framework for which the conclusions of the evaluation will be taken into account; and (ii) knowledge transfer (capacity building) to the Member States.

The proactive approach implemented in the last 2 years of the 2014-2020 programming period provides reassurance that the development of the JASPERS initiative and its development will be closely scrutinised and monitored by the parties involved.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

Directorate General Regional and Urban Policy, Decide Planning: PLAN/2017/2128.

2.Organisation and timing

Tender procedure launched	December 2017
Contract signed	July 2018
First Steering Group meeting	November 2018
Last Steering Group meeting	March 2021
Number of Steering Group meetings	4
Participating DGs (in addition to DG REGIO)	CLIMA: Climate Action ENV: Environment MOVE: Mobility and Transport SG: Secretary General NEAR: Neighbourhood and Enlargement ECFIN: Economic and Financial Affairs

3.Exceptions to the better regulation guidelines

Due to the specialised topic of the evaluation, the open public consultation was replaced by a targeted consultation of relevant stakeholders. The targeted consultation was organised online, during July – September 2019.

4.Consultation of the RSB (if applicable)

The draft staff working document (SWD) was presented at the Regulatory Scrutiny Board (RSB) meeting on 19 May 2021. A preliminary upstream meeting between DG REGIO and the Regulatory Scrutiny Board took place in July 2018.

5.Evidence, sources and quality

See section 4 of the Staff Working Document.

6.Consultation of the Regulatory Scrutiny Board

The draft SWD was presented at the Regulatory Scrutiny Board (RSB) meeting on 19 May 2021 and the RSB issued a negative opinion on 21 May 2021. The RSB comments were addressed in this final document as indicated in the following table.

RSB comment	Reply
1. The report does not articulate what should have been achieved by the time of the evaluation. It does not sufficiently establish what developments can be attributed to JASPERS as compared to other factors.	The general aim of the JASPERS initiative, as defined by the 2006 Memorandum of Understanding, has been to support the implementation of Cohesion Policy by increasing the resources available for project preparation and by improving the quality of technical advice available to project promoters, thus contributing to the preparation of high quality projects. In the 2014-2020 programming period, whose first five years are addressed by this evaluation, the general objective of the JASPERS initiative continued to be the improvement of the quality of investments supported by EU Funds. The evaluation finds that the JASPERS general objective has been broadly achieved, as regards all of its specific objectives with the exception of the transfer of technical knowledge to MS in view of reducing their reliance on JASPERS advisory service. This has been included in the conclusions of the evaluation. With respect to the causal links, the evaluation includes several econometric analysis developed by REGIO and detailed in section 4.3 of Annex 4 of the SWD. For analyses requiring the separation of JASPERS’ contribution from other factors for a given result, the regression analysis was used. This is the case, for example, in the analysis of the correlation between JASPERS’ assistance and the duration of preparation of projects. Nevertheless, the contribution of JASPERS cannot be dissociated in all cases from the other factors that might influence the analysis such as the legal framework in place or the administrative set-up of the major projects approval process.
2. The report does not adequately assess the EU added value and relevance of JASPERS services. It does not examine the potential of JASPERS conflicting objectives and the risk of crowding out capacity building in Member States, and does not draw clear policy lessons from JASPERS’ failure to meet its national capacity building objective.	JASPERS’ EU added value, as indicated by the stakeholders engaged with JASPERS support, primarily lies in the technical expertise brought by its experts who have EU-wide experience acquired while working in many countries and with the European Commission. This has been highlighted in the conclusions of the SWD. The evaluation analyses the relevance perspective of JASPERS during the 2014-2018 period. The needs analysis made by JASPERS before the 2014-2020 period is briefly described in page 4 of the SWD. Because of limitations described in section 4 of the SWD, the relevance of the intervention is assessed mainly in terms of its scope as compared with the previous programming period, while the connection with identified needs is examined in a limited way. The matter of technical capacity building in MS has been thoroughly analysed in the SWD. The conclusion was that during the 2014-2018 evaluation period, while a large majority of activities for capacity building were completed and achieved their objective of delivering training to the Member States, they did not prove sufficient to trigger a visible ripple effect on the technical capacity of Member States that would allow to address issues such as cost-benefit analysis, climate change and environmental requirements in project preparation. A detailed analysis of the findings that led to this conclusion is presented in section 5 – Effectiveness of the SWD and in section 5.2 of Annex 4 to the SWD. The evaluation, however, found examples of a pragmatic approach to capacity building, in the form of ‘training the trainers’ at local level as a source of continuing knowledge dissemination or coordinating training on specific issues across countries. This is highlighted in the conclusions of the SWD. Furthermore JASPERS’ assistance is seen as relevant in the current and future context of the green and digital transition and increased funding opportunities for MS, for which the preparation of mature and high quality projects is fundamental. At the EC level, the need for technical assistance services linked to the future project pipeline has been recognized through the creation of the InvestEU Advisory Hub in which JASPERS will be embedded.
3. The efficiency analysis is not clear enough on observed costs and benefits of JASPERS. The potential for efficiency improvement is not clearly set out.	JASPERS primary objective was to ensure the submission of better quality projects, in view of faster approval thus reducing the time resources of the EC and MS related to their assessment and leading as well to an earlier start of the implementation of investment on the ground. The optimization of costs at project level was an additional outcome of JASPERS’ involvement. As mentioned in the section on limitation, before 2017, data on the actual person-days spent for each assignment by the JASPERS experts was not recorded. A more complete dataset for annual actual work per assignment over the whole period would have been needed for a full estimation of the scope for a more optimal allocation of resources. This is why the analysis of the efficiency criterion is made based on proxy analysis, considering the available data, and it focuses on: -The administrative closure of assignments; -The work intensity distributed across assignments (starting from 2017); -The correlation between human and financial resources and the number of assignments. We have further highlighted in the conclusion section the limitations that have influenced the analysis of the efficiency criterion.
4. The report is not clear how and why it uses the available evidence from different sources.	The SWD has been revised in order to include references, where relevant to the data sources. The use of the various sources has been further explained in section 4. Methods of the SWD.
5. The conclusions do not take a critical view of the limitations and shortcomings identified, to inform policymaking.	The conclusions section has been revised in order to clarify the implications of the identified limitations and shortcomings of the evaluation.

The recommendations made by the RSB in terms of what to improve in the text of the SWD have been taken into account as follows:

RSB – what to improve	Reply
1. The report should clarify what JASPERS should have achieved by 2018. The report should more clearly analyse the new elements added to JASPERS in the 2014-2020 period. For instance, to what extent has the broader scope of JASPERS brought EU added value?	The SWD further clarified the general aim of the JASPERS initiative, as defined by the 2006 Memorandum of Understanding - to support cohesion policy implementation by increasing the resources available for project preparation and by improving the quality of technical advice available to project promoters, thus contributing to the preparation of high quality projects. It specified that in the 2014-2020 programming period, of which its first five years are addressed by this evaluation, the general objective of the JASPERS initiative continued to be the improvement of the quality of investments supported by EU Funds. For the 2014-2020 programming period, JASPERS provided a new kind of support in the form of reviewing the quality of new major projects submitted for EU funding, as required by the regulations, with the aim of ensuring consistency in the assessment and reducing the time taken. This is explained in pages 7-9 of the SWD. The scope of JASPERS has been extended in the 2014-2020 period to cover most of the MS, as shown in Figure 1 included in the SWD. The SWD further clarifies that JASPERS’ EU added value, as indicated by the stakeholders engaged with JASPERS support, primarily lies in the technical expertise brought by its experts who have EU-wide experience acquired while working in many countries and with the European Commission. This was done in the SWD in the analysis and conclusions related to the EU added value criteria.
2. The report should substantiate the relevance of JASPERS beyond the increased demand for its free services. Given the limited development of Member States’ capacity building over the years of JASPERS support, the report should critically assess the continued relevance of this objective and whether JASPERS is the most appropriate support to achieve it. The report should assess if JASPERS’ free services act as a disincentive to develop own capacity in the Member States.	The evaluation analyses the relevance perspective of JASPERS during the 2014-2018 period. The needs analysis made by JASPERS before the 2014-2020 period is described in page 4 of the SWD. JASPERS assistance remains relevant in the current and future context of green and digital transition and increased funding opportunities for MS, for which the preparation of mature and high quality projects is fundamental. At EC level, the need for technical assistance services linked to the future project pipeline has been recognized through the creation of the InvestEU Advisory Hub in which JASPERS will be embedded. The matter of technical capacity building in MS has been thoroughly analysed in page 23-24 of the SWD. The conclusion was that during the 2014-2018 evaluation period, while a large majority of activities for capacity building were completed and achieved their objective of delivering training to the Member States, they did not prove sufficient to trigger a visible ripple effect on the technical capacity of Member States that would allow to address effectively issues such as cost-benefit analysis, climate change and environmental requirements in project preparation. The possible explanations are included in the SWD (“This may be due to high turnover of staff in national authorities administrations, as noted also by the ECA report, but also due to the fact that often JASPERS interacted with consultants contracted by MS for the preparation of project documentation and therefore the learning effect for the staff of national authorities is limited.”) and are not seen as being linked to the free nature of the JASPERS’ services. It is deemed appropriate to mention though that the consultations with stakeholders in the context of the evaluation took place at a moment when the 2021-2027 cohesion policy package was still under negotiation, therefore a certain attempt by the MS for influencing its outcome may not be excluded. The SWD further indicates pragmatic approaches (such as “train the trainers”) that have been implemented during the period of evaluation and that could be potentially replicated in the future in relation to technical capacity-building activities.
3. The efficiency analysis should provide a clearer, comparative review of costs and resources in relation to the benefits achieved. The report should discuss to what extent the increase in human resources in JASPERS has been proportionate to the benefits of the programme. It should also identify areas where there is potential to optimise the allocation of resources. The report should explain how JASPERS has managed to adapt its personnel capacity to changing workloads and whether there is a greater need for flexibility.	Please see the reply provided for point 3 of the table above referring to the RSB comments.
4. The evaluation should be clear about where causal links can be established between the observed developments and JASPERS services. It should be transparent where this is not possible, and discuss what other factors may have played a role.	Please see the reply provided for point 1 of the table above referring to the RSB comments.
5. The report should present and reference more clearly what evidence stems from the external study and the contribution of the inforegio report (annex 4) to fill in the gaps. It should clarify the status of the inforegio report and why it was needed. Some information in annex could be brought forward to the report, in order to have a clearer picture of the evidence base. Where relevant, the report should disaggregate the data to provide a more nuanced analysis.	The SWD has been revised in order to include references, where relevant to the data sources. The use of the various sources has been further explained in section 4. Methods of the SWD. The content of the analysis has been further complemented with additional details from the annexes.
6. The conclusions should be fully aligned with the evidence and be clear about uncertainties.	Please see the reply provided for point 5 of the table above referring to the RSB comments.
7. Given the delay to complete the report, there is scope to briefly explain its usefulness in policy making and informing JASPERS’ development in the period 2021-2027. The report could outline any changes introduced after the period covered by the evaluation and explain whether its results and findings can still form a basis for decision making.	The changes introduced after the period covered by the evaluation and the usefulness of the report for policy making and informing JASPERS’ development in the period 2021-2027 have been detailed in section 7 of the SWD.

Annex 2: Stakeholder consultation

The consultation strategy envisaged two main consultation activities as follows: targeted online consultation and a seminar with stakeholders.

Taking into account the very technical character of JASPERS as a technical assistance initiative available mainly to public bodies, an open public consultation was not required. Yet, a targeted consultation has been organised so as to consult broad range of stakeholders (Managing Authorities, beneficiaries, institutions (EIB), experts and academics,…) that have been involved in JASPERS process and/or have some knowledge of the initiative.

Online targeted consultation

The online targeted consultation was launched on 3rd June 2019 and closed after 17 weeks, on 30th September 2019. The closure was postponed with respect to the initial agreement (closure after 12 weeks) in order to widen, as much as possible, the group of respondents, also by taking advantage of the seminar held on 19th September 2019 during which the online consultation was further disseminated.

The target group comprised stakeholders directly or indirectly involved in JASPERS activities within the different mandates and types of assignments were included. They could have been either direct beneficiaries of JASPERS support (i.e. project promoters in the case of advisory support for major project preparation or REGIO major project team in case of a post-submission appraisal report) or stakeholders indirectly involved in JASPERS activities (i.e. private consult-ants preparing feasibility studies for project promoters and receiving comments by JASPERS or independent experts hired by JASPERS to develop some technical content for the delivery of guidelines or training sessions). In principle, the online consultation was designed for the following stakeholders:

•EU staff (e.g. REGIO major project team and geographical desks, NEAR, INEA);

•National coordination units (both Member States and IPA countries) for the use of EU funds;

•Managing Authorities;

•Project beneficiaries (e.g. local authorities, service providers);

•External consultants and stakeholders involved in project preparation and implementation.

The questionnaire was made available in all official languages of the EU and uploaded on an online platform managed by the external consultant engaged in the mid-term evaluation of the JASPERS initiative. The replies to the survey were collected in an Excel database.

Amongst the total of 554 accesses registered to the link at which the online targeted consultation was available, the online targeted consultation collected a total of 210 “completed” questionnaires, whereas the other 344 questionnaires were considered invalid either be-cause they were empty (283) or completed only partially (61).

Figure 1. Total number of respondents by country

Source: Mid-term evaluation of the JASPERS initiative 2014-2018, COWI (2021)

Note: The total number of respondents considered is 200 (9 European Commission Officials and one multi-country respondent were excluded)

All intended categories of stakeholders have been reached. National public authorities and service providers/private sector represent the main institutions covered by the online consultation, followed by regional and local public authorities and other types of respondents. Most of respondents identified themselves as project promoters or beneficiaries (45,71%) or as Managing Authorities (29%). A lower percentage, around 11,43% of the total number of respondents, was represented by Consultancies providing technical expertise. Another 13% of respondents, instead, does not fall under these categories and identified themselves as European Commission Officials, Cooperating institutions, Intermediate Bodies.

In general, respondents declared to be familiar with one or two types of JASPERS services, while only a small percentage of them stated to be familiar with all types of JASPERS services (11%).

Regarding the frequency of interaction/involvement with JASPERS/JASPERS activities in the period 2014-2020, most of respondents taking part in the survey declared to have interacted with JASPERS on a frequent (63%) or occasional (25%) basis. Only 12% of them, instead, benefitted from JASPERS only once.

In terms of topics covered by JASPERS activities, most of respondents had experience on the subjects of cost-benefit analysis and environmental issues and, to a lesser extent, also with risk and sensitivity analysis and the review of application for co-financing. Conversely, State aid and public procurement result to be the areas of which respondents have lower knowledge.

In terms of effectiveness, stakeholders appreciated positively the role of JASPERS in strengthening their administrative capacity (67%) and in improving the sectoral and strategic planning (61%).

With respect to the relevance of JASPERS services, the majority of respondents expressed the need for JASPERS services for increasing the quality of the projects (97%), for ensuring the compliance with EU environmental directives (90%), for developing the skills related to cost-benefit analysis (88%). Most of the consulted stakeholders (86%) opinionated that JASPERS is also needed for speeding up the approval process of the projects.

In the stakeholders’ view, JASPERS services are suitable to respond to their needs – the main subjects expressed as needs were also selected by stakeholders when they positively assessed JASPERS ability to respond to these needs. As well the majority of stakeholders were satisfied with the services provided by JASPERS especially when it comes to improving the general project documentation (85%) and the technical details in particular (83%).

In relation to external coherence, as alternative advisory services, most of the stakeholders are familiar and made use of private consultancy services (73%) while a third of them used also other advisory services of the EIB.

In terms of areas where JASPERS services provide added value in comparison with other advisory services, more than 80% of stakeholders indicated the technical expertise, the hands-on experience, the possibility of knowledge transfer between Member States and for ensuring the alignment with EU requirements.

86% of the Managing Authorities and 92% of the beneficiary/project promoters indicated that despite improvements in internal capacity, there are still areas where JASPERS support is needed. One of the most important areas where JASPERS might contribute in the future is ensuring that national strategies are aligned to EU sectoral policies (for 70% of Managing Authorities and 89% of beneficiaries/project promoters). It should be noted though that less than a third of the MAs and beneficiaries are prepared to pay for JASPRS support using own resources for technical assistance.

Seminar with stakeholders

As part of the contract signed for the “Mid-term evaluation of the JASPERS initiative in 2014-2020”, a seminar took place at the DG REGIO premises in Brussels on 19th September 2019 with the goal to pre-sent the evaluation’s interim findings and in order to collect feedback and insights from stakeholders. The seminar was organised in two sessions: a first part focused on “Interim findings from the Mid-term evaluation” and a second one on “Future perspectives”.

Nearly 80 participants took part in the seminar to a least one of the sessions. Most of them were Managing Authorities (34) or Project beneficiaries (9) representing a total of 16 Member States. A number of JASPERS experts (13) and European Commission Officials (around 20) also contributed to the discussion during the morning and the afternoon session. The latter were mainly from DG REGIO, but also DG ECFIN, DG ENV, DG CLIMA and DG MOVE were represented.

During the first session, the discussion of the findings was structured per type of service provided by JASPERS:

-Technical advisory: the open debate highlighted several positive elements of JASPERS advisory support such as their extensive hands-on experience, smooth cooperation, facilitating the relation of the Member State or beneficiary with the Commission, capacity building role, independence, and focus on quality (regarded as instrumental for countering politically-driven projects). The establishment of a strong relationship between JASPERS and the authorities of the Member States was identified as a key driver, since continuity in support generates efficiency, as well as synergies in combining strategy preparation with a strong project pipeline. Large consensus was found over the importance of involving JASPERS at an early phase in the project cycle and, more generally, on the need for a continuation of JASPERS advisory function.

-Review function: the stakeholders opinionated that, in terms of internal coherence, a trade-off should be managed between the necessary alignment among Technical Advisory support and the JASPERS IQR Division on one side and the independence of JASPERS IQR division on the other side. In fact, the two JASPERS functions should not dialogue with each other in principle, but at the same time convey the same messages to national authorities and beneficiaries. Overall, involving JASPERS IQR division is perceived by some Member States as a must for ensuring a smooth approval by the Commission. Its effectiveness in documentation improvement was particularly acknowledged as a crucial benefit deriving from appraisal assignments.

-Capacity building and horizontal/strategy support: the Member States expressed a general satisfaction with JASPERS’ capacity building activities and horizontal/strategy support. JASPERS’ ability to prepare useful tools that facilitate the tasks of national counterparts and become part of their toolkit emerged as a significant advantage. Train-the-trainers courses, according to Member States’ experience, were effective in achieving knowledge transfer amongst authorities and experts’ motivation was boosted by this approach. The suggestion emerged for JASPERS to be more proactive and to organise conferences where stakeholders share examples of both good and bad practices, problems and challenges from different projects and countries.

During the second session, each MS attending the seminar expressed their needs regarding the future services to be provided by JASPERS. As an overall input, it was indicated that traditional sectors still require support from JASPERS. At the same time, attendees voiced the wish for a strengthening of JASPERS capacities in non-traditional sectors (e.g. broadband and smart development in general). In addition, Member State representatives emphasised the need to ensure a smooth transition between programming periods.

The findings that emerged from the public consultation were integrated in the evaluation and triangulated with the finds from the quantitative analysis and the qualitative data collected via individual interviews with stakeholders.

Annex 3: Methods and analytical models

Data

The data collected and analysed for this evaluation encompassed a variety of sources as follows:

1)JASPERS’ JADE dataset of assignments for 2014-2020

Provided by JASPERS for the evaluation, this dataset includes 1124 assignments, of which 1096 are for the current programming period and the remaining assignments are for preparation for the next programming period 2021-2027. This data includes details on the main characteristics of an assignment, such as type of service, dates of creation and completion/ closure (if applicable), status by end December 2018, sector, and country. For some of the ESIF major projects submitted to the Commission, the dataset includes also the project reference code in the Commission’s SFC2014 system, although not on a systematic basis. For this reason, we matched JASPERS’ assignments with major projects submitted to the Commission based on checks performed using data and documents from the Commission’s SFC2014 system.

2)European Commission SFC2014 database for ESIF major projects 59

SFC2014 is the system used by the Commission and the Member States for the management of the ESIF funds in the programming period 2014-2020. For the evaluation we used the SFC2014 data for all major projects notified or submitted to the Commission until October 2019 60 . This dataset includes 320 major projects, and it was used as a basis for matching the major projects with JASPERS’ assignments in JADE. The SFC2014 data informs us on the financing proposed or approved for the major projects submitted, the timetables of projects, as well as a series of other project details included in the project application for EU financing. Importantly for the evaluation, the SFC2014 system stores all the documentation submitted by the Member States for these projects, including also the PSA reports and the action completion notes (ACNs) issued by JASPERS’ advisory for assisted projects submitted for Art. 102.2, and the IQR reports issued by JASPERS IQR for projects assessed for the procedure based on Art. 102.1.

3)Assessments and approval of ESIF major projects

For the evaluation of JASPERS’ review function, from the Commission’s SFC2014 system we collected structured data from all the available (66) PSA reports issued by JASPERS IQR for the Commission and all (66) Commission’s observation letters for all projects submitted for Art. 102.2 and interrupted during the approval process. We also screened all (55) IQR reports issued by JASPERS for major projects notified for Art. 102.1. The PSA and IQR reports are also the main basis on which we matched the projects submitted to the Commission and JASPERS’ corresponding advisory and review assignments in JADE.

4)Implementation of ESIF major projects

From the Commission’s SFC2014 system we collected data on the financial and physical implementation from the 2019/2018 annual implementation reports of 166 major projects approved over the reference period. This dataset includes projects assisted and not assisted by JASPERS advisory.

5)JASPERS’ support for ESIF non-major projects

For the ESIF non-major projects assisted by JASPERS advisory, we collected structured data from all (49) available action completion notes, project fiches, and value added fiches provided by JASPERS.

6)CEF projects approved in the current period

For the analysis of JASPERS’ CEF mandate, we used a dataset with all approved CEF projects until August 2019 in order to match JASPERS’ assignments with these projects, and compare them with non-assisted projects. Overall, this dataset includes 265 CEF projects. In addition, DG MOVE provided for the evaluation also the final report of the implementation of JASPERS’ CEF1 mandate.

7)IPA projects approved in the current period

For the analysis of JASPERS’ IPA mandate we matched the IPA assignments with the IPA projects approved over the reference period. The dataset of all IPA projects approved was provided by DG NEAR and it includes 253 projects assisted and not assisted by JASPERS.

8)Planning of assignments

For the analysis of the planning of assignments for JASPERS’ portfolio we collected data on the evolution of these assignments over time from 33 quarterly country action plans for three countries sampled for this evaluation (Croatia, Poland, Romania). The country plans were provided by JASPERS.

9)Actual work hours per assignment

For the analysis of efficiency of JASPERS’ services, we analysed a dataset of 190 assignments created and completed during the period 2017-2019 and for which JASPERS had started recording the actual working time per assignment. The dataset was provided by JASPERS.

10)JASPERS’ human and financial resources

For the analysis of human and financial resources in the section on efficiency, we collected structured data on JASPERS’ annual budgets and staff over the period 2007-2018 from JASPERS’ annual reports published online.

11)JASPERS’ completed assignments for 2007-2013

Dataon assignments completed by JASPERS for the period 2007-2013 was collected from JASPERS’ online annual reports 2010-2018. On this basis, we could identify the assignments by type of service, sector, and country.

12)Performance of JASPERS activities 2007-2013

For the comparison of evaluation findings for the current period with equivalent findings from the period 2007-2013, we collected comparable data from the previous evaluation of JASPERS carried out by DG REGIO in 2012.

13)Major projects approved by the Commission for 2007-2013

SFC2007 is the system used by the Commission and the Member States for the management of ESIF funds during the programming period 2007-2013. From SFC2007 we used the data on major projects approved by the Commission for ERDF and Cohesion Fund for the period 2007-2013. This dataset includes 767 major projects, with data on the EU and total funding for the projects, sector, country, fund, JASPERS assistance, and use of framework contracts for project appraisals.

14)JASPERS’ Network Platform online

From JASPERS’ Networking Platform online we collected data on all training and workshop activities included in the annual programmes starting with 2013 and until 2018. In addition, we also collected the data on all publications published by JASPERS’ in the online library of the platform.

15)Stakeholder’s perceptions of JASPERS’ activities

For this evaluation, we conducted an online targeted consultation with stakeholders with direct experience with JASPERS’ services, including Managing Authorities, projects promoters/ beneficiaries, professionals from consultancy companies, staff of the European Commission, etc. Overall, 210 stakeholders responded to this consultation 61 .

Furthermore, through an external contract, we also explored more in-depth the implementation of JASPERS’ services based on 87 assignments sampled from JASPERS’ portfolio 62 , and on stakeholders’ interviews from 6 countries sampled for the evaluation (Croatia, Italy, North Macedonia, Poland, Romania, and Slovakia). We used these additional elements as background information for the evaluation.

Finally, for the evaluation, we organised also a seminar with stakeholders of JASPERS’ services, where we discussed the interim findings of the evaluation and perspectives on the future of JASPERS with representatives from 15 Member States - all with experience with JASPERS’ services. The seminar was held in September 2019, in Brussels 63 .

Methods

As regards the methods of analysis, we used extensively statistical analysis of the data collected for the evaluation, with an emphasis on median values and distributional patterns. Whenever necessary for the comparison of projects assisted by JASPERS with projects not assisted, we tested the statistical significance of differences in median values for assisted and non-assisted projects by using the Wilcoxon rank-sum test (also known as the Mann–Whitney two-sample statistic). We used this method to understand, for instance, whether the median number of observations in Commission’s interruption letters for projects assisted by JASPERS is lower than the number of observations in interruption letters for non-assisted projects.

In addition, for analyses which required separating JASPERS’ contribution from other co-founding factors for a given result, the evaluation used regression analysis. This is the case, for example, in the analysis of the correlation between JASPERS’ assistance and the duration of preparation of projects. As regards the regression models used, depending on the type of the dependent variable, three types of models were used, as follows:

·linear regression analysis with robust standard errors when the dependent variable is continuous

·logistic regression models when the dependent variable is binary (1/0)

·ordered logistic regression models when the dependent variable is discrete with ordered values (ordinal).

Annex 4: Inforegio report “Analysis and assessment of Services and Performance of JASPERS initiative 2014-2018”

Analysis and assessment of services and performance of JASPERS initiative 2014-2018

May 2021

Disclaimer: The information and views set out in this document are those of the author and do not necessarily reflect the official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this report. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for the use which may be made of the information contained therein.

EUROPEAN COMMISSION

Directorate-General for Regional and Urban Policy

Directorate B — Policy Directorate

Unit B2— Evaluation and European Semester

E-mail: REGIO-EVAL@ec.europa.eu

European Commission
B-1049 Brussels

Table of Contents

1.Introduction

1.1 Purpose and scope

2.JASPERS initiative and its intervention logic

2.1 Description of JASPERS and its objectives

ESIF Mandate

CEF Mandate

IPA Mandate

2.2 JASPERS’ intervention logic

2.3 Baseline and points of comparison

Baseline: programming period 2007-2013

Points of comparison

2.4 Previous evaluations and audits

3.JASPERS’ portfolio of activities during the period 2014-2018

4.Evaluation criteria, data and methods

4.1 Evaluation criteria and issues evaluated

4.2 Data

4.3 Methods

4.4 Robustness and limitations of findings

5.Evaluation findings and supporting evidence

5.1 RELEVANCE

Overall portfolio

ESIF Mandate

CEF Mandate

IPA Mandate

Stakeholder consultation

5.2 EFFECTIVENESS

ESIF Mandate

CEF Mandate

IPA Mandate

Stakeholders consultation

5.3 EFFICIENCY

Planning and management of assignments

Human and financial resources

5.4 COHERENCE

Coherence with EU priorities

Coherence with other advisory services at EU level

Internal coherence between the advisory and review functions

5.5 EU ADDED VALUE

6. Conclusions

JASPERS’ portfolio of services

Technical advisory for ESIF major projects

Technical advisory for ESIF non-major projects

Technical advisory for CEF projects

Technical advisory for IPA projects

ESIF review function

Capacity building and horizontal and strategic support

Stakeholders’ views on the relevance and effectiveness of JASPERS’ services

Efficiency of planning and work allocation

External and internal coherence

EU added value

REFERENCES

Annex 1: Replies to the online targeted consultation

Annex 2: Data and regression models

Annex 3: JASPERS SUPPORT IN 2007-2013

Annex 4: JASPERS SUPPORT 2014-2020

1.Introduction

1.1Purpose and scope

JASPERS (Joint Assistance to Support Projects in European Regions) is a joint initiative for technical advisory of the European Commission and the European Investment Bank (EIB) 64 . JASPERS was effectively launched in 2006 with the main objective to help the new Member States use the available EU funding better and faster. Over a period of fourteen years, JASPERS has grown to be one of the largest initiatives for technical assistance supported by the European Commission, with services available to all Member States and pre-accession countries, and in all sectors relevant for EU investments.

The objective of this evaluation is to assess the implementation of JASPERS’ activities and its performance during the first five years of the current programming period 2014-2020. The evaluation is designed according to the European Commission’s Better Regulation guidelines, and it is structured around the five main evaluation criteria: relevance, effectiveness, efficiency, coherence, and EU added value.

As regards the scope of the evaluation, it covers all JASPERS’ activities implemented until end 2018 across all Member States and sectors. These activities are assessed also in the broader context of the decision making procedures for the approval of projects submitted for EU funding for the Commission services relevant for JASPERS activities.

The structure of this evaluation report is as follows. Next, in Chapter 2, we explain the main coordinates of JASPERS activities and the logic of intervention of its services, as relevant for the current programming period. The chapter also includes a presentation of the main baseline and points of comparisons for the evaluation and a summary of the main findings of other assessments of JASPERS published before this evaluation. In Chapter 3, we present the JASPERS’ portfolio of activities for the current programming period, as accumulated over the reference period for the evaluation 2014-2018, and its evolution over these years. Chapter 4 explains, by evaluation criterion, the main issues analysed in the evaluation, the data collected as evidence, and the methods used for the analysis. This chapter concludes with our assessment of the robustness and limitations of the evaluation findings. The evaluation findings and the supporting evidence are presented in Chapter 5, with sections dedicated to each of the five evaluation criteria. Additional evidence supporting the evaluation findings is included in the annexes to the document. Finally, Chapter 6 includes a discussion of the main results of the evaluation and the conclusions drawn on this basis.

2.JASPERS initiative and its intervention logic

2.1Description of JASPERS and its objectives

Since the launch of the initiative and until 2013, JASPERS worked on providing an independent and free-of-charge advisory service to national authorities in EU Member States and pre-accession countries for the preparation of quality projects co-financed by the European Regional Development Fund (ERDF) and the Cohesion Fund. This activity was carried out under the ESIF mandate of the Directorate General of Regional and Urban Policy (DG REGIO) in the European Commission. The mandate included also activities for capacity building and horizontal and strategic support.

In the current programming period 2014-2020, relative to 2007-2013, JASPERS’ activities expanded both in scope and volume. In terms of scope, JASPERS carries out its core task of technical advisory for project preparation for additional two mandates supported by the Directorate General for Mobility and Transport (DG MOVE) for Connecting Europe Facility (CEF) 65 projects, and Directorate General for European Neighbourhood Policy and Enlargement Negotiations (DG NEAR) for the Instrument for Pre-Accession Assistance (IPA II) 66 assignments. Activities for capacity building and horizontal and strategic support are also included in the scope of these two new mandates. In addition, a novelty for the current period is JASPERS’ review function through which it provides services for the assessment of major projects to Member States and to the Commission in the context of the ESIF mandate. In terms of volume, during the first five years of 2014-2020, JASPERS started more assignments than for the entire period 2007-2013.

JASPERS activities in the current period are based on the Framework Partnership Agreement 67 established in 2014 between the European Commission and the EIB for the ESIF mandate, and subsequently amended in 2015 to include the CEF mandate, and in 2016 for the IPA II mandate.

In organisational terms, JASPERS is organised in 7 divisions, of which 5 are thematic (for Roads, Rail, air, and maritime, Water and wastewater, Smart development, and Energy and solid waste). The two other divisions are for the Independent Quality Review (IQR division) and the Networking and Competence Centre (NCC division) 68 . In addition, JASPERS includes a Quality Management Unit. JASPERS’ head office is in the EIB in Luxembourg, and JASPERS IQR division is located in Brussels. Its thematic divisions are located across regional offices in Bucharest, Sofia, Vienna and Warsaw, and the teams managing the different assignments can combine people from different regional offices depending on their expertise and availability.

In the following paragraphs we present the main coordinates for the three mandates and services provided by JASPERS in the current period.

ESIF Mandate

From the operational perspective, JASPERS’ ESIF mandate is based on annual Specific Grant Agreements (SGAs) signed between the Commission and the EIB for the implementation of JASPERS’ activities. These agreements define the scope and priorities of JASPERS’ activities, the conditions for implementation and the annual resources allocated.

JASPERS’ ESIF mandate includes four main types of services : 1) technical advisory for preparation of major projects ; 2) technical advisory for preparation of non-major projects ; 3) assessment of major projects through the review function; and 4) activities for capacity building and horizontal and strategic support. In addition, in the context of 1) and 2), JASPERS can also provide assistance for implementation of projects. Services of type 1), 2) and 4) are continued from the previous period, while the review function is new.

As regards technical advisory for ESIF major projects 69 , the objective of JASPERS’ assistance is to help Member States in preparing such projects better and faster in order to access EU funding. JASPERS provides this service at the request of the Member States, and it carries out the activity with the agreement of the European Commission. Advisory for major projects is available for all Member States, with priority to less developed and transition regions.

In the current programming period, applying for EU funding for a major project entails detailed preparation of project technical specifications, a feasibility study, a cost benefit analysis and an environmental impact assessment. The project application summarizes not only the technical, economic and financial feasibility of the project documented in the above studies, but also aspects related to the alignment of the project with the objectives of EU Cohesion Policy, and with the EU requirements and standards regarding environmental impacts, adaptation and mitigation measures for climate change, and EU competition rules 70 . An application for EU financing for a major project is subject to Commission’s approval.

In this context, the role of JASPERS advisory is to provide assistance on any of these aspects of project preparation, if and when required by the Member State. While working on an assignment for advisory of major projects, JASPERS issues a series of documents, some procedural and some for guidance addressed to the project beneficiary. Among the procedural documents, the most important type of document that we used extensively for this evaluation is the Action Completion Note (ACN). In an ACN JASPERS describes the project assisted, its contribution to project preparation during the advisory work, and it identifies also significant or other type of issues still outstanding with the project quality at the time when the assignment is completed. As regards the guidance documents, these are notes issued by JASPERS during the advisory work which serve to guide the project beneficiary in the preparation of the project.

JASPERS’ technical advisory for non-major projects is organised in a similar manner and with similar objectives 71 . According to the SGAs, the advisory for non-major projects should prioritise EU countries with fewer major projects such as Cyprus, Estonia, Latvia, Lithuania, Slovakia, Slovenia, and Malta. Further, the SGAs mention that non-major projects can be assisted also in other Member States if they are pilot projects of relevance to EU policy objectives and/or projects which could be replicated on a much wider scale by national authorities. From the operational perspective, upon the request from a Member State and with the agreement of the European Commission, JASPERS opens an assignment for the project and issues similar documents (such as guidance notes and ACNs) as for the major projects. In contrast to the major projects, however, the applications for EU financing for non-major projects are assessed and approved by the Managing Authorities in the Member States.

With regard to the review function, it is designed to assist the Member States and the Commission with assessments of whether the major projects prepared fulfil the quality criteria and the regulatory compliance required for EU funding as follows.

For the programming period 2014-2020, at the Commission proposal, the legislators provided for a new system for the appraisal and approval of new major projects funded from the EU Cohesion Policy. This system entails an independent quality review of the major projects applying for EU funds, and it is organised in two main procedures. First, the procedure based on Art 102.1 in the Common Provisions Regulation (CPR) 72 requires the Member State to notify to the Commission the application for a major project accompanied by an Independent Quality Review performed by independent experts 73 . A second possibility for access to EU funding for major projects is provided for by Art 102.2 in the CPR. For this option, the Member State needs to submit the full project documentation and the project application for the Commission’s assessment and approval 74 , 75 .

In line with these regulatory procedures for the adoption or approval of new major projects, JASPERS’ review function includes two main strands: a) the Independent Quality Review (IQR), organized as a service provided to the Member States for the assessment of projects subsequently notified for Commission’s adoption based on CPR Art. 102.1, and b) the Post Submission Assessment (PSA) service provided to the Commission for support with the assessment of projects submitted for Commission’s approval based on CPR Art 102.2.

The main differences between the two types of assessments include the following. For the IQR, the Member State submits all project documentation directly to JASPERS IQR division for assessment. The IQR division in JASPERS uses the same methodology for project appraisal as the advisory division. Upon assessing the project, however, JASPERS IQR issues its own opinion whether the project meets the required quality criteria and confirms its feasibility and economic viability. The IQR report includes an assessment covering all aspects of the project documentation and application and it is sent to the Member State upon completion. If the IQR conclusions are positive, the Member State notifies the project application together with the IQR report to the Commission for approval based on Art. 102.1.

For the procedure based on Art 102.2, the Member State submits the full project documentation together with the application to the Commission since, in this case, it is the Commission who assesses the quality of the project submitted. Upon receiving the project documentation and the application for EU financing, the Commission asks JASPERS IQR division to assess all aspects of the project, except for alignment of the project with the operational programme, state aid, environmental issues, and the completeness of project application. JASPERS IQR division issues an opinion on project quality in the form of a PSA report sent to the Commission. When compared to an IQR report, a PSA report has a different structure in that it is less comprehensive, and it distinguishes between critical and non-critical issues identified for project quality and documentation for the topics covered 76 , 77 .

Finally, as regards the capacity building and horizontal and strategic support, these are activities which aim at strengthening the capacity for project identification, preparation and implementation in the Member States, as well as at providing support for the preparation of national and regional strategies and plans necessary for the implementation of EU Cohesion Policy. From the operational perspective, the process of assignment creation and implementation is similar as for the advisory function. Also for this type of service, JASPERS issues ACNs when finalising the assignments.

CEF Mandate

The Connecting Europe Facility (CEF) conducted two pilots with small budgets for experimenting with JASPERS’ assistance to the Member States, and for which DG MOVE signed with JASPERS two Specific Grant Agreements (SGAs) in 2015 and 2018. The scope of the CEF mandate includes advisory work for preparation and implementation of projects, as well as advice on horizontal issues relevant for one or more projects, or for one or more countries.

JASPERS started its work with CEF projects under a first pilot mandate (CEF1.1) carried out between July 1st, 2015 and March 31st, 2018. This first CEF mandate had the objective to provide JASPERS’ technical assistance for preparation of eligible projects with the intention to be submitted to CEF calls for the Cohesion envelope of EU financing 78 . Countries and projects to be assisted were identified based on a list of priority projects annexed to the first Specific Grant Agreement (Annex I).

For the period 2018-2020, DG MOVE decided to continue working with JASPERS for a second pilot mandate (CEF1.2). In order to improve the promotion of JASPERS in all Member States and to facilitate its access, in this case the entry point for JASPERS’ advisory for CEF projects was the European Investment Advisory Hub (EIAH) 79 . The technical assistance under this mandate was available for CEF projects applying for CEF Blending calls 80 , with a particular focus, though not limited to, cohesion Member States. Therefore, this second mandate expanded the geographical scope of assistance to all Member States and focused more on an innovative form of financing (blending).

From the operational perspective, in a similar manner as for ESIF projects, JASPERS accepts assignments for support to preparation of CEF projects requested by Member States and upon the agreement of the European Commission. When a CEF assignment is finalized, JASPERS advisory issues a completion note called Project Development Report (PDR).

IPA Mandate

JASPERS’ IPA mandate for the current period functions on the basis of the Specific Grant Agreement (SGA) signed with DG NEAR in 2016. Its advisory work for IPA projects aims at providing technical support for all phases of preparatory work for these projects. Technically, JASPERS has two mandates for the IPA countries : the IPA II mandate covering all pre-accession countries for 2016-2020 81 , and the Serbia National Programme agreement for 2016-2019. For more flexibility in narrative, however, we refer to both mandates as the “IPA mandate” for the current period.

According to the SGA for the IPA mandate, JASPERS’ support is to concentrate on the following issues: a) projects identified to be part of the Single Pipeline of Projects (SPP) or by the National IPA Coordinator in the case of Turkey; b) transport and energy projects for Western Balkans 6 Connectivity Agenda; c) preparation of sector strategies and project pipelines for IPA II; and d) know-how transfer (such as cost benefit analysis, environmental impact assessment etc.).

From the operational perspective, JASPERS accepts an assignment for the IPA mandate upon the request from the beneficiary country and with the agreement of the European Commission. When finalizing the assignments, JASPERS advisory issues action completion notes (ACNs).

Finally, for all these three mandates, JASPERS formalizes its work on assignments in rolling country action plans maintained throughout the period. According to JASPERS’ Quality Manual 82 , the country action plans include working arrangements signed with the beneficiary countries and the list of projects and areas where JASPERS is likely to be asked for support. These plans are updated in line with the requests for support received from the beneficiary countries.

In the following section we present JASPERS’ intervention logic defined by main type of service.

2.2JASPERS’ intervention logic

The main coordinates of the intervention logic of JASPERS’ activities include the type of activities, deliverables, and expected results and impacts. We start with the intervention logic for the services of technical advisory support presented in Figure 2.2.1

Figure 2.2.1: Intervention logic for JASPERS’ advisory support

For advisory services, the main specific objective of JASPERS’ initiative is to enable better preparation of the projects intended for EU financing, thus contributing to their faster approval and to the more general goal of achieving a better implementation of EU funds. Therefore, the immediate results of the assignments delivered are expected in terms of better projects prepared with the assistance of JASPERS and submitted for approval of EU financing, and of timely approval of these projects.

In case of ESIF major projects, timely approval is reflected in a smooth decision process in the Commission, without interruptions due to quality issues. For CEF projects, on the other hand, results are reflected by the submission of the projects to CEF competitions. Similarly, for IPA projects, results are indicated by projects’ approval for IPA financing.

Next, the intervention logic for JASPERS’ review function for the ESIF mandate is illustrated in Figure 2.2.2 below, distinguishing between the two main types of quality assessments: IQR and PSA.

Figure 2.2.2: Intervention logic for JASPERS’ review function

The specific objective of JASPERS’ review activities is to provide an expert independent assessment of the quality of a major project submitted for EU financing from ERDF or Cohesion Fund. As explained above, in the case of IQR, the review is more comprehensive as JASPERS IQR needs to vouch for the project economic and financial viability and its compliance with EU requirements and standards. In the case of the PSA review, on the other hand, JASPERS IQR supports the Commission with its expert view on the more technical aspects of the project and its economic and financial viability.

In both cases, however, the immediate expected result is a timely approval of projects of good quality, and the more general objective is better implementation of EU funds.

For capacity building and horizontal and strategic support, the intervention logic is presented in Figure 2.2.3.

Figure 2.2.3 : Intervention logic for capacity building and horizontal and strategic support

Through these activities, JASPERS’ specific objective is to assist the beneficiary country in the identification, preparation and implementation of projects for EU funding through transfer of knowledge. In this process, JASPERS leverage is based on its experience accumulated by working on a variety of projects across sectors and countries, and in close cooperation with the European Commission. In doing so, JASPERS’ services are expected to contribute to enhanced technical capacity in the beneficiary countries – capacity relevant for a better implementation of EU funds due to more effective identification and preparation of projects.

In the remaining sections of this chapter, we present briefly JASPERS’ activities until 2013 (that is, prior to the reference period for this evaluation), main points of comparison in the current evaluation, and the main findings on JASPERS’ past activities and performance from previous evaluations and audits.

2.3Baseline and points of comparison

Baseline: programming period 2007-2013

Over the period 2007-2013, the European Commission invested 132 million euro, representing 71% of the total costs with JASPERS’ initiative over the period. The remaining 29% was covered primarily by the EIB with in-kind support for additional technical expertise, and also by EBRD and KfW. For this period, JASPERS’ work contributed to mobilising more than 32 billion euro ERDF and Cohesion Fund resources through 383 major projects assisted by JASPERS for preparation and subsequently approved by the Commission 83 .

As explained earlier, during the period 2007-2013, JASPERS’ work was based on the ESIF mandate for DG REGIO. For this period, JASPERS completed around 790 assignments for projects for EU Cohesion Policy, and 6 assignments for IPA projects. A proportion of 82% of these activities were for the advisory function for preparation of major and non-major projects and, to a lesser extent, for project implementation. The remaining activities were dedicated primarily to horizontal and strategic support. In Figure 2.3.1 we present the distribution of these activities by sector and main type of assignment.

Figure 2.3.1 Distribution of JASPERS’ activities by sector and type assignment, 2007-2013 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 796 assignments.

Source : JASPERS’ annual reports 2010-2013

On this basis, we learn that the sector of Water and wastewater treatment was predominant in JASPERS’ portfolio, with a third of all assignments for the period, followed by the sectors of Knowledge economy, energy, and waste combined. As regards the type of service, we note that the activities for capacity building and horizontal and strategic support focused to a larger extent on the sectors of Knowledge economy, energy and waste, Water and wastewater, and on multi-sector assignments.

As regards the distribution of assignment by country, Figure 2.3.2 presents the details. The distribution by country indicates that 76% of JASPERS’ activities in 2007-2013 were concentrated in five beneficiary countries: Poland, Romania, Bulgaria, Czech Republic and Hungary. The remaining assignments were distributed across additional 9 Member States and 3 IPA countries.

Finally, for a perspective of implementation over time, we present also the dynamics of assignments in JASPERS’ portfolio reported as completed on an annual basis over the period 2007-2013. Figure 2.3.3 presents the details.

Figure 2.3.2: Distribution of JASPERS’ activities by country and type assignment, 2007-2013 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 796 assignments

Source : JASPERS’ annual reports 2010-2013

Figure 2.3.3 Assignments completed by JASPERS annually, period 2007-2013 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 796 assignments

Source : JASPERS’ annual reports 2010-2013

From this data, we learn that the peak year for completion of assignments was the 2010 (the fourth year of the programming period), with the annual number (160) representing 20% of all assignments completed for the period). Moreover, by the fifth year of the period (i.e. the year comparable with the reference end date for the evaluation of the current period), JASPERS had carried out 68% of all assignments completed in the period 2007-2013.

Points of comparison

As regards the previous period, we compare JASPERS’ activities and performance in the current period relative to the first five years of the programming period 2007-2013 since the current evaluation covers the first five years of 2014-2020. This information is collected from JASPERS’ annual reports for 2007-2013, and from the previous evaluation of JASPERS activities carried out by DG REGIO in 2012. The next sub-section includes further details on this evaluation.

Furthermore, throughout the current evaluation, in order to estimate JASPERS’ contribution to project preparation and the subsequent performance of the assisted projects in the process of approval for EU funding and later in implementation, we compare systematically the developments for the projects assisted by JASPERS advisory with the non-assisted projects.

As regards the review function, the analysis focuses primarily on the project assessments done by JASPERS IQR for the major projects submitted for Art 102.2, and it compares the PSA assessments of JASPERS IQR division with the Commission’s assessments for the same projects, on the one hand, and also with the assessment provided by JASPERS advisory, on the other hand.

2.4Previous evaluations and audits

There are three previous assessments carried out either as evaluations or as audits of JASPERS’ activities over time. These assessments are as follows: the EIB evaluation published in 2010, the REGIO evaluation published in 2012, and the performance audit of the European Court of Auditors published in 2018.

The first JASPERS’ evaluation carried out by the EIB in 2010 covered the first three years (2006-2009) of JASPERS’ existence 84 . In this evaluation, the EIB evaluators reached the conclusion that JASPERS initiative was fully justified in terms of the demand for its services, and that it generated economic benefits. They also found that, as the initiative had been rapidly established, it also proved an ability to adapt to circumstances. Over time, JASPERS acquired a good reputation with its counterparts, with the initiative evolving into a partnership in which the Commission, JASPERS, and the Member States cooperate on issues of common interest for the implementation of EU funds.

Given this background of high relevance for the initiative, however, the EIB experts also identified scope for improvements in JASPERS’ activities and performance. As regards the effectiveness of technical advisory, the evaluation concluded that, while the quality of some projects had been improved by JASPERS, for a majority of the assignments analysed this had not been the case. Similarly, in terms of JASPERS’ support for grant applications, the evaluation concluded on limited effectiveness, primarily due to reduced coherence between JASPERS’ feedback and the Commission’s assessment.

For the effectiveness of capacity building activities, the EIB evaluators found that, as the initiative was mainly focused on projects, the issue of long term capacity building was seen more as a beneficial spin off of project level activities rather than a priority in itself.

Moreover, the EIB evaluation also looked at the efficiency of JASPERS’ operations and found that, during the initial years, JASPERS experienced unclear operational priorities and limited optimisation of the use of resources. As a result, the initiative became involved in many low priority activities. In this regard, the experts recommended that JASPERS should strive for a maximum possible leverage for its advice, targeting the functioning of the system rather than its products – i.e. “educating the trainers, rather than the trainees“.

Next, in 2012, DG REGIO carried out its own evaluation of JASPERS’ activities and performance - an evaluation which covered the previous period until 2011 85 . This assessment focused on the analysis of the use of JASPERS’ services by sector and Member State, the duration, scale and scope of JASPERS’ assistance, its impact on the timeline of major projects, and its impact on administrative capacity in the beneficiary countries. Overall, as regards the advisory function, the evaluators concluded that JASPERS proved to be of substantial value to the Member States for the preparation of projects and applications for EU funding. They also concluded that JASPERS would have a greater impact on project quality if it were involved in project preparation at an early stage.

As regards capacity building and horizontal support, the REGIO evaluation noted the creation of JASPERS’ Networking Platform in 2011 as a useful forum for the dissemination of guidance and other outputs from the horizontal assignments. Nevertheless, while recognising that JASPERS’ work on projects had the potential to enable a greater understanding EU requirements and standards in the Member States, the evaluators also found that there was limited potential for advisory assignments at project level to help strengthen the administrative capacity at country level, especially in Member States with few projects under development. The evaluation concluded that horizontal and strategic support had higher potential for developing administrative capacity provided that it was based on a strategic approach identifying activities that would have a larger impact on project planning and preparation.

Finally, a third and more recent assessment of JASPERS’ activities and performance was provided by the European Court of Auditors (ECA) in 2018 86 . This performance audit covered the first 10 years (2006-2016) of JASPERS’ activities, and it audited four Member States: Croatia, Malta, Poland, and Romania 87 . As regards the effectiveness of JASPERS’ activities, the ECA auditors concluded that JASPERS contributed to a faster project approval and had a positive effect on the quality of projects and their documentation. For the implementation of projects assisted by JASPERS, the ECA auditors found that, while JASPERS’ comprehensive assistance could generally not have an impact on the financial implementation of EU funds, the assisted projects had been less frequently affected by legality and regularity errors during the compliance audit exercises carried by ECA during 2014-2015.

Concerning JASPERS’ activities for capacity building, ECA noted that JASPERS had increased its focus on capacity building activities in the more recent years, but it found no evidence that these activities have delivered in terms of strengthened capacity in the beneficiary countries.

Furthermore, for the administrative set-up and approach of JASPERS’ activities, the ECA audit also identified scope for improvement for a number of issues. First, the ECA auditors found that the role and responsibilities of the main stakeholders should be defined better by calibrating more precisely the working arrangements between JASPERS and the Member States for all types of services, including also the capacity building activities and the horizontal and strategic support. Second, from the organizational perspective, ECA emphasized the need to strengthen the functional independence of JASPERS IQR relative to the advisory division by establishing clear procedures for the IQR processes that would help enhance the impartiality of the review service. Third, as regards JASPERS’ operational planning, ECA auditors concluded that JASPERS did not target its assistance sufficiently as it did not define clear priority areas for its assistance to the Member States and relied instead for the activities to emerge based on the demand from beneficiary countries. Finally, the ECA auditors also concluded that there was a need for a better system to monitor the efficiency of JASPERS’ operations. Overall, they identified scope for improvement as regards the planning, monitoring and evaluation of JASPERS activities.

Following the ECA performance audit of JASPERS, the Commission established an action plan with measures to be implemented starting with 2018. These measures are designed to address the ECA recommendations accepted by the Commission. By January 2019, the measures related to the clarification of role and responsibilities of the main stakeholders, and the working arrangement within the tripartite cooperation between the Commission, JASPERS and the Member States had been implemented fully or were in the course of being finalized. As regards measures implemented for the planning and performance of JASPERS’ activities, their effects are partially reflected in the evidence used for the current evaluation since the overlap between the timing of the evaluation and the period of implementation for these measures is very limited. Nevertheless, whenever relevant, throughout the document we provide details also regarding the latest developments in this regard.

In the next chapter we present the main developments with JASPERS’ activities over the reference period for the evaluation.

3.JASPERS’ portfolio of activities during the period 2014-2018

For the first five years of the programming period 2014-2020, the European Commission invested close to 150 million euro in JASPERS, representing around 81% of the total costs of the initiative. The remaining 19% was covered by the EIB with in-kind support for additional technical expertise.

In terms of projects approved for EU financing, JASPERS’ work contributed to mobilising almost 23 billion euro ERDF and Cohesion Fund resources through 142 major projects assisted for preparation and adopted or approved by the Commission 88 . Further 2.2 billion euro from the CEF cohesion envelope were mobilised with 23 projects assisted by JASPERS for the CEF1.1 mandate, and 82 million euro with 10 IPA projects assisted by JASPERS and approved for IPA funding.

Over this reference period, JASPERS’ portfolio included 1096 assignments for all mandates and services. Figure 3.1 illustrates the distribution of these assignments by mandate and main type of assignment.

Figure 3.1 JASPERS’ portfolio by mandate and assignment type, 2014-2018

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 1096 assignments

Source : JASPERS’ JADE database, December 2018

The largest mandate in JASPERS’ portfolio is the ESIF mandate, with close to 960 assignments, followed by the IPA mandate with around 100 assignments. As regards the type of services, JASPERS’ traditional function of technical advisory represents almost two thirds of the portfolio. Technical advisory covers support primarily for project preparation for all the three mandates.

The activities for capacity building and horizontal and strategic support in JASPERS’ portfolio are mostly not project specific as they focus on horizontal topics and preparation of national strategies and plans. They are distributed primarily across the ESIF and the IPA mandates.

Finally, the review function is specific to the ESIF mandate, and it includes services for project assessment provided to Member States (for IQR assessments) and to the Commission (for PSA assessments).

In Figure 3.2 we include the distribution of these assignments by sector, covering all mandates and all services.

Figure 3.2 : Distribution of JASPERS’ assignments by sector, 2014-2018 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 1096 assignments.

Source : JASPERS’ JADE database, by December 2018

The distribution by sector indicates that assignments for the transport sectors of Rail, air and maritime are the most numerous in JASPERS’ portfolio, representing 33% of all assignments. The majority of these assignments (70%) are for technical advisory for project preparation. The second largest sector in terms of its weight in the overall portfolio is the sector for Water and wastewater treatment, which also has a substantial number of assignments for technical advisory, when compared with the remaining sectors. The multi sector assignments are primarily capacity building activities, including workshops and training activities on topics relevant across sectors (such as cost benefit analysis, climate adaptation and mitigation etc.).

Furher, in Figure 3.3, we illustrate also the distribution of JASPERS’ portfolio across countries.

Figure 3.3: Distribution of JASPERS’ assignments by country, 2014-2018 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice; data covers 1096 assignments

Source : JASPERS’ JADE database, by December 2018

Across countries, until end 2018, JASPERS initiated assignments in 21 Member States and 5 IPA countries. The countries with the largest numbers of assignments are Poland, Romania, and Croatia, representing collectively 52% of all assignments in JASPERS’ portfolio. Poland stands out also for the relatively large number of assignments for the review function – assignments which correspond to major projects submitted to the Commission for approval of EU financing.

As regards the dynamics over time, 76% of assignments were started in 2014 and thereafter, while the remaining 24% were started prior to 2014 - in preparation for the programming period 2014-2020. By December 2018, 51% of assignments had been completed, and 9% administratively closed 89 , while the rest were still ongoing.

In Figure 3.4 below we present the evolution of the ongoing assignments in JASPERS’ portfolio by year between 2014 and 2018.

Figure 3.4: Ongoing assignments by main type by year, 2014-2018 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 1096 assignments

Source : JASPERS’ JADE database, by December 2018

The yearly data for ongoing assignments indicates that, already by the first year of the programming period 2014-2020, JASPERS’ portfolio included more than 300 assignments for technical advisory and for capacity building and horizontal and strategic support 90 .

The assignments for the review function started in 2015, and increased over time as project preparation was completed and projects were submitted for EU-financing.

Moreover, while over time some assignments were completed or closed, we see that the number of assignments ongoing per year increased steadily until 2017. The year 2018 is the first year in the period where the total number of ongoing assignments is lower than in the previous year. By end December 2018, the number of assignments still ongoing was 435. Figure 3.5 illustrates the number of completed assignments on an annual basis over the period 2014-2018.

On this basis, we note that the pace of assignment completion has accelerated significantly starting with year 2017. For the first two years, JASPERS completed assignments for technical advisory and for capacity building and horizontal and strategic support, while the completion of review assignments started with 2017. A detailed analysis of the timeline of all assignments is included in Chapter 5, in the section for efficiency.

Figure 3.5 Completed assignments by main type by year, 2014-2018 (number assignments)

Note: CB = Capacity Building, HSS = Horizontal and Strategic Advice ; data covers 535 assignments

Source : JASPERS’ JADE database, by December 2018

In the following chapter we present the evaluation criteria on which this evaluation is based, together with the data and methods used. The chapter includes also our assessment of the robustness and limitations of the evaluation findings.

4.Evaluation criteria, data and methods

4.1Evaluation criteria and issues evaluated

For this evaluation we assessed JASPERS’ activities and performance during the period 2014-2018 based on the five evaluation criteria from the Commission’s Better Regulation: relevance, effectiveness, efficiency, coherence, and EU added value.

For the criterion of relevance, we took into account the fact that JASPERS’ activities are initiated at the request of Member States, and also the fact that JASPERS accepted all assignments requested by the Member States for the current period. The aspects analysed for this criterion include the structure of JASPERS portfolio emerging during the reference period across mandates, and by type of service, project size, sector, and country, as well as the coverage with JASPERS assistance of projects submitted for EU financing. For capacity building and horizontal activities, we analysed the frequency of topics and issues for which this type of support was requested by the beneficiary countries. Moreover, for all types of services, we analysed stakeholders’ perceptions as regards the relevance of JASPERS activities and their experiences with the services provided.

As regards the effectiveness of JASPERS’ services, we analysed the extent to which and how the objectives of JASPERS’ services for a faster and better preparation of projects were fulfilled. We analysed also the contribution of JASPERS to a more consistent and faster process of assessment and approval of ESIF major projects, and its impact on the financial implementation. Subject to data availability, similar issues were analysed also for all other remaining services of JASPERS. Finally, we analysed also the stakeholders’ perceptions as regards the effect of JASPERS’ on a variety of issues relevant for project preparation and administrative capacity in the beneficiary countries.

For efficiency, the evaluation looked primarily at the efficiency of the planning of JASPERS’ assignments and the rate of administrative closures over the full reference period. We analysed also the work intensity by type of assignment over a period of two years for which data was available. The section on efficiency also includes an analysis of the evolution over time of the financial and human resources dedicated to the initiative in the current period, relative to 2007-2013.

The evaluation criterion of coherence presents the analysis of the extent to which JASPERS’ activities are aligned with and promote the EU requirements and standards for the objectives of EU Cohesion Policy, and with higher level EU priorities for environment, climate change adaptation and mitigation, and EU competition policy. For this criterion we also analysed the internal coherence of JASPERS’ services by comparing the consistency between the project assessments provided by JASPERS IQR in the context of the review function, with the assessments delivered by JASPERS advisory in the context of technical support for project preparation.

Finally, the criterion of EU added value was assessed primarily based on the stakeholders’ perceptions on the added value of JASPERS’ services relative to other type of schemes for technical advisory and capacity building. In this context, stakeholders’ were also asked for their opinions on the need for JASPERS’ services in the future, and on their willingness to pay for these services from own technical assistance.

4.2Data

For this evaluation we collected and analysed data from a variety of sources as follows:

16)JASPERS’ JADE dataset of assignments for 2014-2020

17)European Commission SFC2014 database for ESIF major projects 91

SFC2014 is the system used by the Commission and the Member States for the management of the ESIF funds in the programming period 2014-2020. For the evaluation we used the SFC2014 data for all major projects notified or submitted to the Commission until October 2019 92 . This dataset includes 320 major projects, and it was used as a basis for matching the major projects with JASPERS’ assignments in JADE. The SFC2014 data informs us on the financing proposed or approved for the major projects submitted, the timetables of projects, as well as a series of other project details included in the project application for EU financing. Importantly for the evaluation, the SFC2014 system stores all the documentation submitted by the Member States for these projects, including also the PSA reports and the action completion notes (ACNs) issued by JASPERS’ advisory for assisted projects submitted for Art. 102.2, and the IQR reports issued by JASPERS IQR for projects assessed for the procedure based on Art. 102.1.

18)Assessments and approval of ESIF major projects

For the evaluation of JASPERS’ review function, from the Commission’s SFC2014 system we collected structured data from all the available (69) PSA reports issued by JASPERS IQR for the Commission and all (69) Commission’s observation letters for all projects submitted for Art. 102.2 and interrupted during the approval process. We also screened all (55) IQR reports issued by JASPERS for major projects notified for Art. 102.1. The PSA and IQR reports are also the main basis on which we matched the projects submitted to the Commission and JASPERS’ corresponding advisory and review assignments in JADE.

19)Implementation of ESIF major projects

20)JASPERS’ support for ESIF non-major projects

For the ESIF non-major projects assisted by JASPERS advisory, we collected structured data from all (49) available action completion notes, project fiches, and value added fiches provided by JASPERS for the sampled countries.

21)CEF projects approved in the current period

22)IPA projects approved in the current period

23)Planning of assignments

24)Actual work hours per assignment

25)JASPERS’ human and financial resources

26)JASPERS’ completed assignments for 2007-2013

Data on assignments completed by JASPERS for the period 2007-2013 was collected from JASPERS’ online annual reports 2010-2018. On this basis, we could identify the assignments by type of service, sector, and country.

27)Performance of JASPERS activities 2007-2013

28)Major projects approved by the Commission for 2007-2013 (SFC2007)

29)JASPERS’ Network Platform website

From JASPERS’ Networking Platform website we collected data on all training and workshop activities included in the annual programmes starting with 2013 and until 2018. In addition, we also collected the data on all publications published by JASPERS’ in the online library of the platform.

30)Stakeholder’s perceptions of JASPERS’activities

Furthermore, based on the documents from an external contract, we also explored more in-depth the implementation of JASPERS’ services based on 87 assignments sampled from JASPERS’ portfolio 94 , and on stakeholders’ interviews from 6 countries sampled for the evaluation (Croatia, Italy, North Macedonia, Poland, Romania, and Slovakia). We used these additional elements as background information for this evaluation report.

4.3Methods

As regards the methods of analysis, we used extensively statistical analysis of the data collected for the evaluation, with an emphasis on median values and distributional patterns. Whenever necessary for the comparison of projects assisted by JASPERS with projects not assisted, we tested the statistical significance of differences in median values for assisted and non-assisted projects by using the Wilcoxon rank-sum test (also known as the Mann–Whitney two-sample statistic). We used this method to understand, for instance, whether the median number of observations in Commission’s interruption letters for projects assisted by JASPERS is lower than the median number of observations in interruption letters for non-assisted projects.

In addition, for analyses which required separating JASPERS’ contribution from other co-founding factors for a given result, we used regression analysis. This is the case, for example, in the analysis of the correlation between JASPERS’ assistance and the duration of preparation of projects. As regards the regression models used, depending on the type of the dependent variable, we used three types of models as follows 95 :

·linear regression analysis with robust standard errors when the dependent variable is continuous

·logistic regression models when the dependent variable is binary (1/0)

·ordered logistic regression models when the dependent variable is discrete with ordered values (ordinal).

4.4Robustness and limitations of findings

In our assessment, a very large majority of the findings in this evaluation are robust as they are based on evidence collected mostly at the level of the entire relevant population. In Annex 2, Tables 2.1.1-2.1.3 we summarize the data coverage by evaluation criterion and topic for evaluation, thus identifying the extent to which each topic analysed is covered by representative data.

There are, however, exceptions for topics for which the possibility to collect the necessary data was much more limited. First, this is the case of the analysis of effectiveness of JASPERS’ advisory for non-major projects which remains partial due to lack of follow up data on what happens with these projects once JASPERS’ advisory is completed. JASPERS does not collect this type of data, and to our data requests made to the six Member States sampled for this evaluation, only two of them responded.

Another important topic which is only partially covered by the evaluation refers to the efficiency of the time allocation for JASPERS’ activities. For this topic, we analysed only the assignments completed during a period of two years (2017-2019). In our estimate, however, a more complete dataset for annual actual work per assignment over the entire period would have been needed for a full estimation of the remaining scope for a more optimal allocation of resources. More complete data, however, was not available for the full reference period of the current evaluation as JASPERS started collecting this data in 2017, and only at assignment level.

Finally, a third topic less covered by this evaluation refers to the IQR review service. While for the PSA review service we could conduct an extensive analysis for effectiveness and internal coherence as we collected the necessary structured data from the Commission SFC2014 system and from JASPERS, for the IQR process the scope for analysis was much more limited. We learnt that, during the reference period of the evaluation, the IQR process was less formal than the PSA assessment for the Commission, with much of the information related to the assessment of projects conveyed by email in a non-structured form 96 . As a result, we could not carry out a systematic analysis of the effectiveness and coherence of this service relative to the advisory services comparable with the one for the PSA.

The remaining of this document includes the presentation of the evaluation findings and the supporting evidence, followed by the chapter of conclusions.

Finally, the author would like to thank JASPERS’ staff for their cooperation in sharing data for this evaluation and for their availability for clarifications whenever needed. Furthermore, we are also grateful to our Commission colleagues participating in the Steering Group of the evaluation for their continuing cooperation and support.

5.Evaluation findings and supporting evidence

In this chapter we present the main results of the evaluation of JASPERS initiative by the five evaluation criteria of the Commission’s Better Regulation : relevance, effectiveness, efficiency, coherence, and EU added value. For presentation purposes, we further organised the finding by JASPERS’ mandate and type of function (advisory, review, and horizontal functions) to the extent it proved necessary to distinguish along these dimensions. This applies in particular for the criteria of relevance, effectiveness and coherence, but less so for efficiency and EU added value where the issues analysed are of a more horizontal nature and therefore apply to all mandates.

As regards the presentation of the supporting evidence, the main text includes the evidence that could be presented in a more concise manner for an integrated narrative, while the annexes include further details and disaggregation of the data supporting the conclusions.

5.1RELEVANCE

For the evaluation criterion of relevance we analysed the structure of JASPERS’ portfolio over the reference period 2014-2018 97 , often in comparison with its activities during the previous programming period 2007-2013. Furthermore, the relevance of activities during the current period is also analysed in the broader context of the Commission’s decision making process for the approval of EU financing for projects prepared with the assistance of JASPERS advisory or reviewed by JASPERS IQR.

Given that all JASPERS’ activities are initiated at the request of the Member States, the main evaluation approach applied was to assess the structure of JASPERS’ portfolio which emerged so far. The main issues analysed here include the distribution of assignments requested by Member States by sector and by type of assignment for each mandate, the concentration of support across Member States, the type and timing of support provided by JASPERS in the context of project preparation, the types of project supported, as well as the Member States’ choices between the two procedures for the approval of major projects for the ESIF mandate. The section includes also an analysis of the distribution of the activities for capacity building and horizontal and strategic support.

This section is first organised by JASPERS’ mandate and it concludes with the stakeholder online consultation on relevance of JASPERS’ activities for all mandates. The main findings are summarized in the Text box 5.1.1 below.

Text Box 5.1.1 Main findings for the relevance of JASPERS’ activities 2014-2018

OVERALL

1)In the current period JASPERS remained relevant for the preparation of projects, with its activities mandated by three Commission services and providing more services for a larger number of countries compared to the previous period. In addition, JASPERS has become relevant also for the new function of quality review of ESIF major projects submitted for EU funding.

ESIF MANDATE

2)Overall mandate: Relative to the previous period, JASPERS’ ESIF portfolio has diversified and expanded. With around 960 assignments for the first 5 years of the current period, the ESIF portfolio now includes close to 60% assignments for project preparation (compared with 82% previously), 23% for capacity building and horizontal advice (compared with 18% previously), and the remaining for the new review function.

3)Overall mandate: Across sectors and relative to the previous period, the main changes in the portfolio structure include an increased share for assignments in transport (higher by 16%), and a reduced share for water and wastewater treatment assignments (lower by 11%). The share of assignments for transport is now highest in the ESIF portfolio (50%).

4)Overall mandate: A majority of JASPERS’s activities (57%) remain concentrated in three main beneficiary countries (Romania, Croatia and Poland), compared with 59% of assignments concentrated in three beneficiary countries (Romania, Poland and Bulgaria) in the previous period.

5)Major projects: 58% of projects submitted to the Commission for adoption/ approval were assisted by JASPERS advisory for project preparation, compared with 50% in 2007-2013. Most countries maintained high to very high rates of JASPERS’ assistance for their projects submitted, while most of the non-assisted projects were submitted by Poland (76%).

6)Major projects: Compared to non-assisted projects, the ones assisted by JASPERS advisory tend to be larger in size. This pattern is maintained from the previous period.

7)Major projects: Most frequent topics for JASPERS’ technical advice for the preparation of new projects include the feasibility study and cost and benefit analysis.

8)Major projects: For 42% of new major projects assisted and submitted, JASPERS technical assistance was requested from the start of the project preparation. For additional 50% of the projects, JASPERS assistance started during the stage of core preparation.

9)Non-major projects: The share of assignments for non-major projects in JASPERS’ portfolio for technical advisory has increased from 21% to 25% between the two periods.

10) Non-major projects: Across sectors, main changes in the portfolio structure include an increased share for assignments in transport (higher by 7%), and a reduced share for water and wastewater treatment assignments (lower by 4%). The share of assignments for water and wastewater treatment, however, remains highest in the portfolio (34%).

11) Non-major projects: A majority (74%) of these assignments is concentrated in three main beneficiary countries (Croatia, Romania, Bulgaria). In the previous period, 63% of such assignments were concentrated in three main beneficiary countries (Romania, Poland, Bulgaria).

12) Non-major projects: In almost 70% of assignments, JASPERS analyzed the non-major projects in a similar manner as for major projects since beneficiaries prepared feasibility studies and cost benefit analyses for these projects even in absence of a regulatory obligation.

13) Non-major projects: Most frequent topics for JASPERS technical advice for the preparation of projects include the feasibility study and cost and benefit analysis.

14) Review function: JASPERS’ new review function is highly relevant as it assesses all new major projects submitted under the legislative framework for the period 2014-2020. Of the two procedures (Art 102.1 and Art 102.2), more than 70% of new major projects were assessed with the support of the PSA service provided by JASPERS IQR to the Commission for the procedure based on Art. 102.2.

15) Review function: While there is significant variability across projects and countries, when compared with the procedure based on Art. 102.1, the PSA based procedure (Art 102.2) is more likely to be chosen by Member States for larger projects, for projects submitted before the full year with the Omnibus Regulation adopted (2019), and for projects not assisted by JASPERS advisory.

16) Capacity building and HSS: Across all support activities for technical capacity, while the share of multi-country trainings and workshops on topics of high relevance for all countries has increased, the horizontal and strategic support tailored to country specific needs remains prevalent.

17) Capacity building and HSS: A majority (57%) of activities for capacity building included in JASPERS’ portfolio in the current period are multi-country assignments. The most frequent topics for these training activities include climate change, environment, state aid, and cost benefit analysis.

18) Capacity building and HSS: The demand for country specific capacity building activities was not high during the reference period.

19) Capacity building and HSS: Two thirds of country specific assignments for horizontal and specific support are of horizontal nature. One third supported national strategies and plans.

20) Capacity building and HSS: Close to 60% of country specific horizontal and strategic support is concentrated in 3 Member States (Romania, Poland and Croatia).

21) Capacity building and HSS: More than 60% of activities for capacity building and horizontal and strategic advice were started at the beginning of (or before) the programming period 2014-2020.

CEF1.1 MANDATE

22)CEF1.1: For the first pilot, JASPERS advisory assisted 29 projects, of which 23 were approved for CEF financing for 6 cohesion countries.

23) CEF1.1: Compared with all projects approved for CEF financing, the CEF projects assisted by JASPERS advisory for the first pilot are more concentrated on works (rather than studies), and they tend to be much larger in terms of eligible costs.

24) CEF1.1: In a sample of 9 assignments, frequent topics for advisory include project technical specification, cost benefit analysis, environmental issues, procurement and implementation, and grant application.

IPA MANDATE

25) IPA: Relative to the previous period, JASPERS’ IPA mandate has expanded and diversified in 2014-2020.

26) IPA: Almost three quarters of assignments in JASPERS’ portfolio are for advisory for project preparation, with the largest share (42%) for the sector of Water and wastewater treatment.

27) IPA: More than half of the advisory assignments are concentrated in two beneficiary IPA countries (North Macedonia and Serbia).

28) IPA: Overall, projects assisted by JASPERS advisory and approved for financing represent 7% of all IPA projects approved for financing until 2019. In most of these assignments, JASPERS helps with the guidance and supervision of contracts for technical assistance.

ONLINE CONSULTATION

29) Online consultation: Almost all of the 210 respondents consider JASPERS relevant for support to increase project quality, and for assisting the alignment of projects with EU environmental directives. Further, a large majority perceive JASPERS to be relevant for support with the development of skills for cost benefit analysis.

30) Online consultation: The most frequent reasons for requesting JASPERS’ services are perceptions of increased chances for faster project approval.

31) Online consultation: A large majority of stakeholders think that JASPERS’ has a high ability to respond to needs for improving project quality, for ensuring compliance with EU environmental regulations, and for assisting with the development of skills for cost benefit analysis.

32) Online consultation: Overall, a very large majority of stakeholders rated highly JASPERS’ services in terms of coherence with EU standards, quality of advice, effectiveness and timeliness.

For each of these evaluation findings the supporting evidence is as follows.

Overall portfolio

In the current period JASPERS remained relevant for project preparation, with its activities mandated by three Commission services and providing more services for a larger number of countries compared to the previous period. In addition, JASPERS has become relevant also for the new function of quality review of ESIF major projects submitted for EU funding.

JASPERS’s technical support continued to be highly relevant also in the current programming period 2014-2020. As explained in Chapter 2, JASPERS assistance is now available for all Member States and IPA countries, and it is mandated by three Commission services (DG REGIO, DG MOVE, and DG NEAR). Its services include technical advisory for the preparation of major and non-major projects, activities for capacity building and horizontal and strategic support, and a new review function for projects supported by Cohesion Policy. For the first 5 years of the current period, JASPERS registered around 1100 assignments in 26 countries.

By comparison, in the previous period 2007-2013, JASPERS provided technical assistance for 14 Member States and 3 IPA countries through almost 800 completed assignments. Its services included technical advisory for the preparation of major and non-major projects, activities for capacity building, and horizontal and strategic support for projects supported by Cohesion Policy (see Table A3.2 in Annex 3 for details).

In what follows, we look into more detail at the distribution of JASPERS’ activities by mandate.

ESIF Mandate

Relative to the previous period, JASPERS’ portfolio for the ESIF mandate has diversified and expanded. As regards sectors, the portfolio now also includes a higher share of assignments for transport, and a lower share for water and wastewater treatment.

JASPERS’ portfolio for the ESIF mandate in current period until December 2018 includes close to 960 assignments, of which 59% are for direct technical assistance for project preparation (and to a much lesser extent implementation), 23% for horizontal activities and capacity building, and the remaining 18% for its new function of IQR and post-submission assessments of major projects. Table 5.1.2 presents the details by sector and assignment type.

Table 5.1.2 JASPERS portfolio for the ESIF mandate 2014-2018, by sector and type

Note: TA = Technical Assistance.

Source: JASPERS’ JADE database, December 2018

Half of these assignments were created for projects and issues relevant for the transport sector (Rail, air, maritime, and roads), and 20% for the sector of Water and wastewater treatment.

When compared with the portfolio of assignments completed for th period 2007-2013 by sector, we note primarily an overall increase in the share of assignments for transport by 16%, primarily due to a significant increase in the share of assignments for Rail, air and maritime (from 19% to 32%) between the two periods. As regards the share of assignments for Water and wastewater, it has diminished by 11% (from 31% o 20%) between the two periods. The distribution of JASPERS’ activities by sector and type of assignment for the previous period is presented in Table A3.1 in Annex 3.

A majority of JASPERS’ activities for the ESIF mandate remain concentrated in three main beneficiary countries.

When looking at the distribution of the ESIF assignments at country level we see that 57% of these assignments were carried out in three main beneficiary countries (Romania, Croatia, and Poland), 5% are multi country assignments, and the rest are distributed across 18 more Member States (see Table A4.1 in Annex 4 for details).

By comparison, during the previous period, 59% of JASPERS’ activities for this mandate were also concentrated in three main beneficiary countries (Poland, Romania, Bulgaria), 1% were multi-country assignments, and the rest of activities were distributed across 11 more Member States (see Table A3.2 in Annex 3 for details).

Technical advisory for major projects

A majority (58%) of major projects submitted to the Commission have been supported by JASPERS advisory for project preparation. Most countries maintained high to very high rates of JASPERS’ assistance for their projects submitted, while 76% of the non-assisted projects were submitted by Poland.

As regards the assignments for the preparation of major projects, in order to understand what happened with the projects assisted by JASPERS advisory once the technical assistance is completed, we looked at the data in Commission’s SFC2014 system for major projects submitted and approved until October 2019 (that is, 10 months after the reference end date for JASPERS’ portfolio).

In SFC2014 there are three types of projects: 1) new projects submitted for approval based on CPR Art 102.1 (subject to IQR), 2) new projects submitted for Commission’s assessment and decision based on CPR Art 102.2 (subject to PSA), and 3) projects phased from the previous programming period and submitted based on CPR Art 103. Overall, until October 2019, 184 major projects submitted to the Commission had been assisted by JASPERS advisory for project preparation, implying a rate of coverage of submitted projects by JASPERS technical advisory of 58%. Table 5.1.3 summarizes the number of projects submitted for the 3 types of procedures by sector and JASPERS advisory.

Table 5.1.3 Major projects submitted to the Commission until October 2019 98 , 99

Note : TA = Technical Advisory.

Source: European Commission SFC2014, October 2019

Data in Table 5.1.3 indicates, for example, that 110 of the major projects submitted were for Rail, air and maritime, with 27 of them subject to IQR and notified to the Commission for approval. Furthermore, 67% of the 27 projects subject to IQR in this sector had been assisted by JASPERS advisory for project preparation. The same interpretation applies to all sectors and types of projects presented in the table.

These data indicate that a majority of major projects submitted to the Commission were assisted by JASPERS advisory in all sectors, except for Roads where the rate of assistance is lower (36%). The highest rates of assistance apply to Water and wastewater (80%) and Smart development and ICT (68%).

When compared with the data for projects approved in the previous programming period, while the overall rate of Jaspers assistance for major projects increased (from 50% for 2007-2013 to 58% currently), the relative weight of sectors has also changed. In 2007-2013, the higher rate of coverage by JASPERS assistance was observed for Energy and solid waste (72%), followed by Water and wastewater treatment (63%). For Smart development and ICT, on the other hand, the proportion of assisted projects was 35% in 2007-2013, therefore increasing significantly to 68% in the current period. The distribution of assignments by sector for 2007-2013 is presented in Table A3.4 in Annex 3.

In what follows, among all major projects submitted to the Commission for approval for the current programming period, for this evaluation we focus primarily on the new major projects submitted for Art. 102.1 and 102.2. This is because the phased projects submitted for Art.103 had been prepared during the previous programming period 2007-2013 100 , and therefore are less comparable with the new major projects as regards JASPERS’ services for advisory and project appraisal.

For the two types of procedures for approval/ adoption of new major projects in the current programming period we observe a higher prevalence of assisted projects (80%) among the new projects submitted to IQR, relative to the direct assessment by the Commission (61%). In absolute numbers, however, the number of projects submitted for the PSA based procedure, and which had been assisted by JASPERS advisory, is much higher.

At country level, for the new projects, we find that 14 countries out of 18 used JASPERS technical assistance for the preparation of at least 75% of their new major projects submitted for funding. Table A4.3 in Annex 4 presents the distribution of these projects by country. By comparison, for the previous period, latest data available in SFC2007 indicates that 50% of almost 770 such projects were assisted by JASPERS, with the coverage of assistance ranging from 5% and 100% at country level. This means that, In 10 out of the 14 countries assisted, JASPERS advisory contributed to the preparation of all or almost all their major projects approved by the Commission (see Table A3.3 in Annex 3 for the details for 2007-2013).

Further for the comparison of the two periods in relative terms, we can distinguish the following four situations 101 . First, there are countries which increased the use JASPERS technical assistance for their new major projects submitted in the current period. These include countries with previous experience with JASPERS services such as Bulgaria, Italy, Romania and Slovenia. Second, some other Member States are new to JASPERS support in the current period. These include Austria, France, Spain and Portugal. Third, four Member States (Hungary, Latvia, Lithuania and Slovakia) maintained full support (100%) of JASPERS advisory for all the projects submitted to the Commission in both periods. And fourth, two Member States - Czechia and Poland - used JASPERS advisory to a lesser extent, with a reduction of more than 10% in the number of projects assisted in all projects submitted at least until October 2019.

In absolute terms, however, we find that a large majority of non-assisted new major projects submitted to the Commission belong to Poland. From a total of 210 new major projects submitted by all countries, 53 out of 70 projects not assisted by JASPERS advisory came from Poland (with 43 in Roads, and 10 in Rail, air, maritime). In the previous period the share of Poland in all non-assisted major projects was only 22%, and therefore it increased significantly during the first 5 years of the current period. Moreover, as Poland has reduced its overall use of JASPERS advisory for new major projects submitted to the Commission by 12% relative to the previous period, this data also indicates that the country has increased reliance on own resources for the preparation of projects at least in some sectors. This applies especially for Roads 102 , where 84% of new projects submitted did not require JASPERS advisory in the current period and, to a lesser extent, it also applies to Rail, air and maritime, with 30% of new projects submitted not assisted by JASPERS advisory.

Major projects assisted by JASPERS tend to be larger in terms of median total costs.

As regards the size of projects, we looked at all the 178 new major projects adopted/ approved by the Commission during the reference period and calculated the median values for total cost and EU funding by sector. Table 5.1.3 presents the details.

Table 5.1.3 Median values of major project financing by sector, source and JASPERS advisory

Note: Data refers to new major projects (Art 102.1 and 102.2) approved/ adopted by EC. It covers 178 projects, of which 120 assisted by JASPERS advisory.

Source: European Commission SFC2014, October 2019

When comparing assisted and non-assisted projects, we find that the major projects assisted by JASPERS advisory tend to be larger in terms of total costs in all sectors, with differences significantly higher for the sectors of Roads (329 million euro for assisted projects vs 255 million euro for non-assisted projects) and Energy and solid waste (209 million euro vs 129 million euro) 103 , EU funding maintains the same pattern, with the exception of ERDF and EIB loans for Rail, air and maritime for which the median value of EU grants/ loans is higher for non-assisted projects.

Relative to period 2007-2013 we find that, while the total funding for all major projects tends to be higher in the current period, the same pattern of larger major projects assisted by JASPERS advisory can be observed in both periods. The median total costs and EU contribution by sector and JASPERS’ support for 2007-2013 are presented in Table A3.5 in Annex 3.

Most frequent topics for JASPERS’ technical advice for new projects include feasibility analysis and cost and benefit analysis.

In order to understand the type of advice that JASPERS advisory provides to assisted projects, we analysed the contribution of JASPERS’s support as reflected by action completion notes (ACNs) for completed assignments. We carried out this analysis for 81 new major projects assisted by JASPERS and submitted to the Commission for assessment and approval based on Art 102.2. This sample represents 88% of all such projects in the Commission’s database 104 . The data collected from the ACNs of these projects is summarised in Table 5.1.4.

Table 5.1.4 Projects supported by JASPERS for preparation by topic

Topic	Projects advised for the topic	% in all project analysed

(1)	(2)	(6)
Feasibility study	77	95%
Cost benefit analysis	77	95%
Climate issues	49	60%
Risk and sensitivity analysis	48	59%
Environmental issues	44	54%
Project description	34	42%
Project costs	30	37%
Financial plan	26	32%
Capacity beneficiary	23	28%
State aid	20	25%
Implementation schedule	20	25%
Indicators	14	17%

Note: Data covers 81 major projects submitted for approval based on Art 102.2.

Source: ACNs for projects assisted by JJASPERS advisory.

The classification of topics presented in column (1) of Table 5.1.4 is derived from the structure of the analysis provided by JASPERS in ACNs. For each of these topics, JASPERS provides a detailed overview of the project, followed by a text box with JASPERS’ assessment, comments for improvement, and significant outstanding issues (if applicable). Further, the ACNs also mention frequently the topics on which JASPERS provided technical advice, as well as topics on which the experts provided a review. On this basis, we classified the projects analysed by topic in two main types: 1) projects for which JASPERS’ experts stated that they contributed to the revision on the topic in previous versions of project documentation, or provided comments or signaled outstanding issues at the time when the ACN was finalized; and 2) projects which did not meet the criteria for 1), but for which JASPERS’ experts provided a positive assessment or a description 105 .

In this way we learn that the most frequent topics on which JASPERS contributed to project preparation (type 1) above) include cost-benefit analysis and feasibility studies ( in 95% of the project analysed). For the remaining topics, the frequency of contribution varies significantly across projects, possibly also due to the scope of the initial request for the assignment from the Member State.

For a comparison with the period 2007-2013 we collected the findings on this issue from the previous Commission’s evaluation of JASPERS carried out in 2012. Tables A3.6 in Annex 3 presents the details. On this basis, we note that cost-benefit analysis had been a major topic for assistance for a large majority of assisted projects also in the previous period, while for the remaining topics the extent of JASPERS’s contribution appears more extensive in the current period, as measured by frequency of topics in assisted projects.

For 42% of new major projects assisted and submitted, JASPERS technical assistance was requested from the start of the project preparation. For additional 50% of the projects, JASPERS assistance started during then stage of core preparation.

Further on JASPERS’ contribution, we also analysed the stage of JASPERS’ involvement in project preparation. This analysis covers almost all (99%) of the 140 projects assisted by JASPERS advisory and submitted to the Commission for adoption/ approval. For these projects we compared the calendar duration of the advisory assignments reported in JADE with the timeline for the preparation of projects reported by Member States in their applications for major projects in SFC2014.

On this basis, we classified the advisory assignments as follows: Group 1: assignments created from the start of the core preparatory stage of the project or earlier; Group 2: assignments created after the start but during the core preparatory stage; and Group 3: assignments created after the core preparatory stage. For the definition of the core preparatory stage, we considered the following steps in process preparation: feasibility study, cost benefit analysis, environmental impact assessment and design studies 106 .

Table 5.1.5 presents the number of projects assisted by JASPERS advisory by sector and stage of involvement (i.e. group).

Table 5.1.5: Stage involvement of JASPERS advisory in project preparation

Sector	Number projects	Group 1 (from start)	Group 2 (during prep)	Group 3 (after prep)	% Group 1 by sector
Energy and solid waste	13	3	8	2	23%
Rail, air, maritime	52	23	26	3	44%
Roads	32	5	23	4	16%
Smart development and ICT	18	11	6	1	61%
Water and wastewater	24	17	7	0	71%
Total	139	59	70	10	42%
% in total		42%	50%	7%

Note: Data covers 99% of new major projects assisted by JASPERS and submitted to the Commission.

Source: JASPERS’ JADE database, December 2018; European Commission SFC2014, October 2019

On this basis we learn that JASPERS’ assistance was required from the start of the project preparation in 42% of cases. Across sectors, this was the case especially for projects in Water and wastewater treatment (71%) and for Rail, air and maritime (44%). The sector with the lowest relative share of projects assisted from the beginning of the preparation stage is Roads (16%).

In terms of size of projects, we find that projects assisted from the start of the preparation stage have a median cost of 212 million euro, while the ones for which advisory starts during the preparation stage tend to be smaller (198 million euro). The few projects assisted after the end of the core preparation stage, however, are the largest (315 million euro median total cost).

Further details on the nature of JASPERS’ contribution to supported projects can be inferred by linking the data on stage of involvement with the data on JASPERS’ contribution from the ACNs. Table 5.1.6 below presents the details.

The data in Table 5.1.6 covers 87% of all new major projects assisted by JASPERS advisory and submitted to the Commission for Art. 102.2 (for which we could collect data from ACNs).

On this basis, we learn that for 40% of projects in which JASPERS provided advice on feasibility study, the advisory assignments started from the beginning of project preparation.

Table 5.1.6 JASPERS’ contribution by topic and stage involvement in project preparation

Note: Data covers 80 projects assisted by JASPERS and submitted for approval for Art. 102.2.

Source: ACNs for JASPERS advisory support (for topic) and JADE and SFC2014 (classification by group).

For the remaining topics the interpretation is similar. For the financial plan, climate, and environmental issues, we learn that the advisory support was most frequent for the assignments which started during the preparatory stage.

More generally, however, we also note that advisory support for most topics appears also in assignments started after the core preparation (group 3) – a situation which can be explained if JAPERS was asked to review the project documentation and application. This also indicates that the extent and depth of JASPERS’ advice on topics is likely to differ by stage, possibly with implications on the effectiveness of the technical support 107 . This aspect is analysed further in the section of effectiveness.

Technical advisory for non-major projects

Representing a quarter of advisory support for project preparation, assignments for non-major projects for the sector of Water and wastewater treatment maintain the highest share in the portfolio.

During the reference period, JASPERS advisory provided support also for 140 non-major projects, representing 25% of all assignments for advisory for the ESIF mandate. The non-major projects include all projects which cannot qualify as major projects according to the regulatory definition for ESIF investments, and they are subject to the assessment and approval of Managing Authorities in the Member States. As regards the completion of advisory assignments for these projects during the reference period, more than half of them were already completed or administratively closed by end December 2018.

In terms of distribution by sector, more than one third of the non-major project assisted by JASPERS are in Water and wastewater (34%), followed by Rail, air and maritime (with 23%). Table 5.1.7 presents the details.

Table 5.1.7 Non-major projects assisted by JASPERS TA, by sector and status

Sector \ Status	On-going	Completed and closed	Total	% in total
Energy and solid waste	9	9	18	13%
Rail, air, maritime	21	11	32	23%
Roads	10	3	13	9%
Smart development and ICT	6	24	30	21%
Water and wastewater	18	29	47	34%
Total	64	76	140	100%
% in total	46%	54%

Sources : JASPERS’s JADE database by end December 2018

On this basis, we see that the sector of Water and wastewater treatment maintained the highest share of assignments for non-major projects in both periods.

Relative to the previous period, we see that the structure of the portfolio for non-major projects has not changed much. During 2007-2013, JASPERS assisted most frequently non-major projects in Water and wastewater (38% compared to 34% currently), followed by Knowledge Economy, energy and solid waste combined (33% compared to 34%). Table A3.7 in Annex 3 presents the distribution of assignments for non-major projects by sector for the period 2007-2013.

A majority of assignments for non-major projects is concentrated in three main beneficiary countries.

As regards distribution by country, for the current period, JASPERS’ portfolio indicates that a large majority (74%) of these assignments were carried out in three main beneficiary countries: Croatia, Romania and Bulgaria, while the remaining assignments are distributed across other 11 Member States. Table A4.4 in Annex 4 presents the details.

Therefore, we note that the distribution of JASPERS’s assignments for non-major projects in the current period is highly skewed towards these three main beneficiary countries which are also among the largest beneficiaries for advisory support for major projects. On the one hand, this indicates higher needs for technical support for project preparation for these countries. On the other hand, however, this finding also suggests that the realization of assignments for non-major projects is not fully aligned with the initial expectations and priorities set out in the grant agreements between the Commission and JASPERS. While the grant agreements for the ESIF mandate indicate non-major projects as a priority especially for countries with fewer major projects (CY, EE, LV, LT, MT, SK, and SI), we find that JASPERS assisted only 11 non-major projects in these countries. This is due to the fact that JASPERS advisory services are primarily driven by the demand coming from the Member States, and the demand from the countries with fewer major projects for advisory for non-major projects proved lower than expected initially.

By comparison, in the previous period, JASPERS advisory completed 133 assignments for ESIF non-major projects across 12 Member States, representing 21% of all assignments for project preparation. Also for 2007-2013, a majority (63%) of assignments for non-major projects were concentrated in three countries (Romania, Poland and Bulgaria). Between the two periods, significant reductions in relative shares of such assignments in JASPERS’ portfolio are observed for Slovenia (12%) and Poland (10%). Table A3.8 in Annex 3 presents the distribution of JASPERS’ assignments for non-major projects by country for 2007-2013.

For 70% of non-major projects assisted by JAPERS, beneficiaries in the sampled countries prepared feasibility studies and cost benefit analyses even in absence of a regulatory obligation. Most frequent topics for advisory support include feasibility issues and the cost benefit analysis.

Further for the analysis of JASPERS’ advisory for non-major projects we screened 49 ACNs and closure notes for JASPERS’ assignments for non-major projects completed in 4 Member States sampled for this evaluation: Croatia, Poland, Romania and Slovakia 108 . The assignments analysed represent 74% of the ESIF portfolio for assignments completed for non-major projects 109 .

In terms of size, the median total cost for these projects is 35 million euro, with values ranging from a minimum of 7 million euro to a maximum of 79 million euro 110 . Relatively larger non-major projects in terms of median total cost are assisted by JASPERS in sectors such as Energy and solid waste (54 million euro) and Water and wastewater (49 million euro), while projects assisted in Smart development and ICT have a smaller median cost (19 million euro).

By comparison, for the previous programming period, the REGIO evaluation from 2012 informs us that the average total cost for non-major projects assisted by JASPERS was 30 million euro, with the largest projects assisted in Water and wastewater (average total cost of 35 million euro).

As regards topics of JASPERS’ contribution to the preparation of non-major projects, we learn that there are two types of situations. First, for almost 70% of the projects analysed, JASPERS carried out an analysis similar to major projects, contributing to and/ or reviewing all relevant topics such as project description, feasibility analysis, cost benefit analysis etc. This indicates that for a majority of non-major projects beneficiaries prepare project documentation at similar standards as for major projects, even if the regulation does not require them to do so 111 . Further, when providing assistance for such projects, JASPERS issued an ACN with the same structure as for a major project (labelled long ACN). Second, there are also non-major projects for which JASPERS’ analysis is much shorter, covering broadly the main aspects of the project, and so is the corresponding ACN (labelled short ACN).

As for major projects, we classified the technical assistance according to evidence for JASPERS’ contribution recorded in the advisory notes. We considered that JASPERS provided advice and direct input into project analysis and preparation whenever that was clearly stated in the action completion note for the respective topic, or when JASPERS’ experts made comments or raised significant issues for the topic in question in the final version of the ACN. The results are summarized in Table 5.1.8.

Table 5.1.8 Topics of technical advice for assisted non-major projects

Source: ACNs for 49 completed non-major projects with JASPERS advisory.

Table 5.1.8 reads as follows. For project description, for examples, JASPERS assisted 10 projects for which it issued a long ACNs, and 6 projects for which it issued short ACNs. Overall, the projects for which Jaspers provided support related to project description represent 33% of all projects analysed.

On this basis, we find that, for non-major projects, JASPERS contributes most frequently to topics such as option analysis and cost-benefit analysis (in 60-70% of the projects). Least frequent topics for JASPERS’ contribution are indicators, state aid, project costs and capacity of beneficiaries. As expected, there are also differences in the coverage of technical contribution between projects with long and short ACNs, with significantly more substantive and wider technical support provided for the first type of projects.

In terms of differences in size between these two types of projects, when compared with the projects with long ACNs, the non-major projects with short ACNs tend to have smaller median total costs (up to 18 million euro) in sectors such as Rail, air and maritime and Roads, but larger median total costs (40 million euro) for projects in Smart development and ICT.

Review function

JASPERS’ new review function is highly relevant since it assesses the quality of all new major projects submitted to the Commission under the legislative framework for the period 2014-2020. Of the two procedures, 71% of new major projects were assessed by the PSA service.

For both procedures (Art. 102.1 and Art. 102.2) described in Chapter 2, JASPERS’ contribution is highly relevant for the assessment of major projects since essentially all new major projects notified or submitted for approval of EU financing need to go through one of the two procedures.

Based on JASPERS’ portfolio data, we find that JASPERS carried out 175 assignments for IQR and PSA assessments until December 2018. This data is, however, very dynamic as it changes frequently with new major projects submitted for assessment by Member States. When looking into the Commission’s SFC2014 system for projects submitted 10 months later (in October 2019), for instance, the number of projects with IQR and PSA assessments were already 210. Furthermore, for the PSA service, it is also possible that there are more than one PSA reports for a given projects if the Commission asks JASPERS to issue additional PSA reports for projects revised following interruptions for quality issues during the approval procedure.

Therefore, for the analysis of the review function, we used the data on new major projects submitted or notified from the Commission’s SFC2014 database since it covers a longer period after July 2018 when the revision of the CPR through the Omnibus Regulation was adopted (discussed below). This dataset spans until October 2019, and it includes 210 major projects relevant for JASPERS’ review function.

In Table A4.3 in Annex 4 already mentioned above we summarize the number of new major projects submitted by Member States to the Commission for the two procedures for adoption/ approval. This data indicates that, of all 210 new projects in the Commission database by October 2019, 71% had been submitted for the Commission’s assessment and adoption based on Art 102.2, thus implying PSA assessments from JASPERS. The remaining 29% of projects approved by the Commission were assessed based on the IQR process provided for by Art. 102.1 procedure.

While there is significant variability across projects and countries, the PSA based procedure is more likely to be chosen for larger projects, for projects submitted before the full year with the Omnibus Regulation (2019), and for projects not assisted by JASPERS advisory.

At country level we can identify two main practices. First, there are Member States who seem to have a preference for one of the two services. Until October 2019, countries which used only the IQR service included Croatia, Germany, Latvia, Lithuania and Malta. For Art 102.2 procedure, countries which submitted projects only directly for Commission’s assessment included Austria, Greece, Hungary, Portugal, Slovenia and Spain. Second, there are also Member States which chose to use both possibilities for the assessment of their new major projects. These include: Bulgaria, Czechia, France, Italy, Poland, Romania and Slovakia.

The choice of one of the two procedures for the assessment of the major projects is in the remit of the Member State. One possible reason for this choice refers to practical aspects of implementation. Prior to the revision of the CPR with the Omnibus Regulation in July 2018 112 , Member States could certify expenditure for major projects submitted based on CPR Art 102.2 as soon as the Commission determined the completeness of project documentation and application in SFC2014. In contrast, if the project was first submitted to JASPERS IQR, for certification, the Managing Authority would need to wait until JASPERS’ assessment was complete and the project application was submitted to the Commission. Given the time difference between the two processes, our conjecture is that this might have provided an incentive to Member States to favor the procedure based on CPR Art 102.2 and trigger the use of the PSA service in the process 113 .

With the Omnibus regulation, however, this incentive was balanced between the two procedures, so that now, also for projects subject to IQR, there is the possibility to certify expenditure as soon as JASPERS IQR determines the completeness of project documentation and application. In Table 5.1.9 we present the submission of major projects by type of procedure (102.1 or 102.2) and over time.

Table 5.1.9 New major projects by year of submission and procedure

Year submission	Art. 102.1 (IQR)	Art. 102.2 (PSA)	Total	% of total
2015	0	14	14	7%
2016	3	19	22	10%
2017	13	51	64	30%
2018	17	44	61	29%
2019	27	22	49	23%
Total	60	150	210	100%
% in total	29%	71%	100%

Source: European Commission SFC2014, October 2019

This data indicates that the number of projects submitted for the procedure based on Art. 102.2 was systematically higher until 2019 – a first year which shows a reversed emerging trend. Therefore, this preliminary evidence for the first full year with the Omnibus Regulation adopted seems to point towards a more balanced use of the two procedures by the Member States.

Nevertheless, there could be also other factors that influence the decision to use one of the two procedures for submission of major projects, such as the characteristics of the projects and the experience of the Managing Authority or the beneficiary with JASPERS’ services. For this reason, we ran a logistic regression model in order to analyse the correlation between the use of the two procedures and factors such as: use of JASPERS advisory assistance for project preparation, the size of the projects in terms of total costs or EU amounts, year of submission relative to year 2019, and sector. This analysis covers 96% of all major projects submitted to the Commission for the two procedures with complete data for all variables in the model. The results from these models are included in Table A2.2.2 in Annex 2.

From this analysis we learn that the likelihood of the use of the procedure based on Art 102.2 (and PSA) increases for larger projects and for projects submitted before 2019. Furthermore, we also find that projects with JASPERS advisory in preparatory stages are more likely to be submitted for Art 102.1 (IQR), and so are also the projects prepared for sectors such as Rail, air and maritime and, to a lesser extent, Water and wastewater 114 .

Capacity buildings and horizontal and strategic support

Next, for the ESIF mandate, we look at JASPERS’ activities for capacity building and horizontal and strategic support. While all the activities aim at supporting the beneficiary countries in the identification and preparation of projects for EU investments, in the evaluation we distinguish between two main types of activities: trainings and workshops on relevant topics (classified as capacity building activities), and hands-on assignments supporting beneficiaries and Managing Authorities in carrying out more horizontal and strategic tasks such as preparation of national plans and strategies providing the framework for the EU investments. In practice, all these types of activities aim at supporting relevant technical capacities in beneficiary countries.

Across all support for technical capacity, while the share of multi-country training and workshops on topics of high relevance for all countries has increased, the horizontal and strategic support tailored to country specific needs remains prevalent.

The activities for capacity building and horizontal and strategic support were maintained in JASPERS’s mandate also in the current period. Based on JADE data, we find that JASPERS has carried out 220 such assignments until December 2018, with 75% of them for horizontal and strategic advice.

By comparison, in 2007-2013 JASPERS completed close to 140 such assignments, with 96% for horizontal and strategic advice. Therefore, we note an increase in the relative share of assignments for capacity building to from 4% to 25% between the two periods.

These assignments can be tailored to the needs of a specific country, or can be organised with a multi country perspective for topics of high relevance to several or all Member States. In Table 5.1.10 we present the classification by type of assignment and audience.

Table 5.1.10 Assignments for capacity building and horizontal and strategic support

Type assignments	Country specific	Multi country	Total
Capacity building	23	31	54
Horizontal and strategic support	159	7	166
Total	182	38	220
% in total	83%	17%	100%

Source : JASPERS’s JADE database, 31 December 2018

Starting with capacity building activities, in contrast to the previous period when almost all assignments were country specific, we see that there has been an increase in the frequency of multi country assignments in the current period. JASPERS explained that this increase in multi-country activities was enabled by the launch of the Networking Platform in 2013.

A majority (57%) of activities for capacity building included in JASPERS’ portfolio in the current period are multi-country assignments. The most frequent topics for these training activities include climate change, environment, state aid, and cost benefit analysis.

Of all 54 assignments for capacity building in JASPERS’ portfolio for 2014-2020, 57% are multi-country assignments. By December 2018, 21 of these assignments were completed and are relevant for an analysis of their impacts on the capacity for project preparation in the Member States. In addition, we found 3 multi country horizontal assignments which are very similar in nature to the ones for capacity building, and therefore we include them in the analysis. Finally, from the online website of JASPERS’ Networking Platform, we identified other 14 multi-country workshops and events organised by JASPERS during the period 2014-2018 - assignments which are in addition to the assignments recorded in JASPERS’ database. Overall we collected a set of 38 multi-country assignments completed over the reference period. Table 5.1.11 presents a classification of these assignments by topic.

Table 5.1.11 Multi country assignments for capacity building

Topic	Number assignments	Years when organised
Climate change and environment	9	2012, 2014-2018
State aid	5	2014-2016
Cost benefit analysis	6	2012, 2015-2018
Urban development	5	2016, 2017
Projects in energy sector	2	2014-2015
Other topics	11	2012-2018
Total	38

Source : JASPERS’ JADE database, December 2018 and JASPERS’ Networking Platform online, September 2020

The most frequent multi-country trainings/ workshops were organised for topics related to climate change and environment, and for state aid. Workshops for cost benefit analysis and urban development are the next most frequent types of workshops organised, while for the remaining topics JASPERS organised about one event each topic over the period 2012-2018. About 62% of these activities were carried out at the very beginning of the programming period (or before its start) until year 2015.

The demand for country specific capacity building activities was not high during the reference period of the evaluation.

As regards the country specific assignments for capacity building, 19 out of 23 assignments were completed or still on-going by December 2018 in 10 Member States. More than 60% of these assignments are dedicated to climate change adaptation and environment, with the remaining assignments created mostly for horizontal aspects and planning and implementation.

From the analysis at country level carried out for the five Member States sampled for this evaluation, however, we learn that the demand for this type of tailored support for capacity building was rather limited during the reference period. These countries include Croatia, Italy, Poland, Romania and Slovakia, and they account for 43% of all such assignments.

From Poland, for instance, we learnt that, while the country sends numerous participants to multi-country trainings and workshops for capacity building, for tailored support it has identified only climate change adaptation as a relevant topic and requested one assignment. Similarly, Slovakia (with 2 assignments) did not engage in significant country specific capacity building, partly due to challenges they experienced with staff turnover. Like Poland, however, also Slovakia participated frequently in multi country activities for capacity building. Further, Croatia had only one request for this type of support, while Italy did not use it at all. The only country in the sample with a larger number of assignments for country specific support was Romania (with 5 completed assignments), which also opted for tailored support primarily for environmental issues and climate change adaptation.

Two thirds of country specific assignments for horizontal and specific support are of horizontal nature. One third supported national strategies and plans.

As regards the horizontal and strategic support, 132 out of 159 assignments had been completed or were still on-going by December 2018. Of these, 95% are country specific assignments. The classification of these assignments by main type and sectors, determined based on their titles, is presented in Table 5.1.12 below.

From this data we learn that the assignments supporting national strategies, masterplans and plans represent 32% of all country specific assignments. The remaining 68% were created for a variety of horizontal tasks, including support for analysis of sector level investments, guidelines for CBA and other sector specific topics, preparation of project pipeline, preparation of operational programmes and other horizontal tasks. Therefore, on this basis, we conclude that this type of country specific support is predominantly horizontal and focuses relatively less on strategic support.

Table 5.1.12 Country specific assignments for horizontal and strategic advice

Objective support	Assignments	Sectors/ topics
Support national/ sectoral plans	22	8 Transport, 8 Urban mobility; 3 Waste; rest for other 3 sectors
Support national strategies	18	7 Transport; 3 Smart specialisation; 3 Wastewater; rest for other 4 sectors
Capacity building	14	3 Multiple funding schemes; 2 Disaster risk management; rest for other 9 topics
Support for sectoral analysis	13	6 Energy, 2 Transport, 2 Urban development; rest in other 3 sectors
Guidelines	11	5 CBA; 4 Water and wastewater; rest for other 2 topics
Project pipeline preparation	10	3 Energy efficiency; 2 Waste; rest for other 5 sectors
Operational programme preparation	9
Other	28
Total	125

Note: Data covers assignments completed or on-going by December 2018. “Other” refers to a variety of topics such as ex-ante conditionalities, benchmarking, transport models, etc. 115 .

Source: Authors’ classification based on assignment titles in JASPERS’ JADE database, December 2018.

Close to 60% of country specific horizontal and strategic support is concentrated in 3 Member States (Romania, Poland and Croatia).

At country level, we find that close to 60% of country specific assignments are concentrated in three Member States: Romania, Poland and Croatia, while the remaining assignments are distributed across other 12 countries. Table A4.5 in Annex 4 presents the distribution of these assignments by country.

More than 60% of activities for capacity building and horizontal and strategic advice were started at the beginning of (or before) the programming period 2014-2020.

Over time, 64% of all assignments for capacity building and horizontal and strategic advice were completed during the period 2012-2015 since many were organised in view of the preparation for the new programming period 2014-2020. In our view, this helps explain also the prevalence of the horizontal support, as many of these assignments focused on the preparation of operational programmes, identification of project pipeline, preparation for fulfilment of ex-ante conditionalities, sectoral analyses for planning investments etc.

Furthermore, in addition to the activities recorded in the JASPERS’ database, we also looked at JASPERS’ Networking Platform online 116 . Created in 2013, the Networking Platform includes information on JASPERS’ past and upcoming events, annual programmes starting with 2013 to date, and a library for JASPERS guidance and capacity building documents. The library includes around 60 publications, with more than 60% of them covering topics such as cost benefit analysis, knowledge economy, state aid, and waste management. Similarly, as with the activities, a large majority of these publications (75%) were published during 2012-2014, in preparation for the upcoming programming period.

CEF Mandate

For JASPERS’s CEF mandate we distinguish between the two mandates (CEF1.1 and CEF1.2) described in Chapter 2, with the evaluation looking only at the first mandate.

Technical advisory for CEF projects

For the first pilot (CEF1.1), JASPERS advisory assisted 29 projects, of which 23 were approved for CEF financing.

Based on the final annual report of CEF1.1 mandate, we learn that JASPERS carried out 31 assignments in support of preparation of 29 CEF projects, 117 with a large majority of them in the rail sector. Table 5.1.13 presents the details.

Table 5.1.13 JASPERS’s assignments for CEF1.1 mandate

Sector	Assignments	CEF projects assisted
Rail	25	23
Roads	5	5
Inland waterways	1	1
Total	31	29

Source : Final Report for the CEF1.1 mandate

At country level, the initial agreement for the first CEF pilot (CEF1.1) was that JASPERS would support projects chosen from the list of 58 projects in 7 cohesion countries included in Annex I of the Special Grant Agreement. The actual number of projects supported was determined by the budget available for the pilot for JASPERS assistance 118 . Overall, the projects supported by JASPERS by the end of the first CEF pilot represents 55% of the projects listed as potential candidates in the SGA for CEF1.1. They were submitted for 3 CEF calls for the cohesion envelope during 2014-2016, and 23 of them were approved for CEF financing.

As regards the contribution of this first pilot to the overall portfolio of CEF projects at the EU level, we find that the projects assisted by JASPERS represent approximately 11% of all the 265 CEF projects approved for cohesion and blending actions until August 2019. Out of 14 cohesion Member States with projects approved for financing, 6 requested JASPERS’ technical support for project preparation in proportions ranging between 11% and 47% of the number of CEF projects at country level. Table A4.7 in Annex 4 presents the distribution of CEF projects and JASPERS’ advisory for CEF by country.

Compared with all projects approved for CEF financing, the CEF projects assisted by JASPERS advisory are more concentrated on works (rather than studies), and they tend to be much larger in terms of eligible costs.

In terms of type of projects, 81% of projects supported by JASPERS support are for works or mixed (works and studies), compared with 64% for the CEF overall project portfolio.

As regards project size, we calculated the median total costs and CEF funding for all the projects included in the Commission’s database by August 2019. Table 5.1.14 presents the details by type of project while also distinguishing between projects assisted and not assisted by JASPERS advisory.

Table 5.1.14 Size of CEF projects assisted by JASPERS, relative to non assisted (median values)

Type project	Actual eligible costs (million euro)		Actual CEF funding (million euro)
	Projects assisted by JASPERS	Projects not assisted by JASPERS	Projects assisted by JASPERS	Projects not assisted by JASPERS
Works	114	35	97	12
Works + studies	65	29	55	22
Studies	4.4	3.7	3.7	3.2

Note : data covers projects approved for CEF cohesion and blending actions (26 projects assisted, and 239 projects not assisted).

Source : DG MOVE data on CEF projects approved, August 2019

On this basis, we learn that, relative to the ones not assisted by JASPERS, the projects assisted by JASPERS tend to be much larger in terms of median values of actual eligible costs and actual CEF funding for actions for works and mixed (works plus studies). For actions including only studies, the median costs and EU grants are similar for the two groups.

Furthermore, from Jaspers we learnt also that 6 assignments for studies out of a total of 15 were dedicated to feasibility studies prepared for investment projects intended for CEF financing 119 .

In a sample of 9 assignments, frequent topics for advisory include project technical specification, cost benefit analysis, environmental issues, procurement and implementation, and grant application.

In order to understand the type of support provided by JASPERS advisory to the assisted projects, we analysed the project development reports for a sample of 9 CEF projects for works assisted 120 . We classified JASPERS input to these projects as a contribution to the project on a given topic if there was any evidence in the assignment document of advice or comments on the topic. For a majority of these projects, JASPERS’ main contribution related to technical specifications of the project, cost-benefit analysis, environmental issues, procurement and implementation aspects, and the grant application.

As regards the CEF1.2 mandate started in 2018, the timing of the evaluation overlaps only with a few months into this mandate, and therefore there is limited scope for an assessment.

Furthermore, in addition to the assignments for technical advisory, the JADE database includes also two assignments for horizontal advice, but the number of is too limited to draw conclusions for an evaluation.

IPA Mandate

Technical advisory for IPA projects

Relative to the previous period, JASPERS’ IPA mandate has expanded and diversified in the period 2014-2020.

For the IPA mandate in the current period, JASPERS’ portfolio includes 102 assignments, compared with 6 assignments completed for IPA in the previous period. The IPA mandate now covers all types of JASPERS services, except for the review services which apply only to the ESIF projects.

Almost three quarters of assignments in JASPERS’ portfolio for the IPA mandate are for advisory for project preparation. Projects for Water and wastewater represent 42% across sectors.

JASPERS’ IPA portfolio is distributed in 72% assignments for support to preparation of projects, and 22% for activities for capacity building and horizontal and strategic advice. The remaining assignments provide support for the implementation of projects. The distribution of these assignments by sector is presented in Table 5.1.15.

Table 5.1.15 JASPERS’ portfolio for the IPA mandate 2014-2018, by sector and type assignment

Sector	Project preparation	Capacity building and horizontal	Implementation	Total	% total
Energy and solid waste	21	2		23	23%
Rail, air and maritime	15	7	6	28	27%
Roads	4		1	5	5%
Water and wastewater	33	10		43	42%
Multi sector		3		3	3%
Total	73	22	7	102	100%
% total	72%	22%	7%	100%

Source : JASPERS’ JADE database, December 2018.

While the sector of water and wastewater remains dominant for the IPA portfolio, for transport projects we note a higher proportion of assignments for Rail, air and maritime relative to Roads. We found no assignment for the sector of smart development and ICT in the IPA mandate until December 2018.

More than half of the advisory assignments are concentrated in two beneficiary IPA countries.

At country level, across 5 countries with advisory support over the reference period 121 , more than half (53%) of assignments for project preparation are concentrated in North Macedonia and Serbia. For Montenegro, for example, project preparation was more limited, with 13 out of 17 assignments concentrated on capacity building and horizontal support. Table A4.8 in Annex 4 presents the distribution of assignments for the IPA mandate by country.

Overall, projects assisted by JASPERS advisory and approved for financing represent 7% of all IPA projects approved for financing until 2018. In most of these assignments JASPERS helps with the guidance and supervision of contracts for technical assistance.

In order to understand what happened to the projects after being assisted by JASPERS advisory for preparation, we matched the data from JASPERS’ portfolio with the data on approved projects from DG NEAR. The data on IPA projects from DG NEAR refers to projects approved and started during 2011-2019 122 . Overall, this dataset includes 253 projects, of which 17 could be matched with JASPERS’ assignments for the current period. This data is summarized in Table 5.1.16, showing also the distribution by type of IPA project.

Table 5.1.16 IPA projects started during 2011-2019

Type IPA project	All projects	JASPERS assisted
Investment facility	4
Investment grant	33	3
Technical assistance	216	14
Total	253	17

Source : DG NEAR dataset, August 2019

Therefore, on this basis we learn that the IPA projects assisted by JASPERS’ advisory represent 7% of the total number of IPA projects approved in the current period.

As regards the type of advice provided by JASPERS advisory to IPA projects, it depends on the type of the project. As reflected in the dataset from DG NEAR, we learn that JASPERS can advise projects for investment grants, but also for supervision of technical assistance contracts for studies or other preparatory steps in view of investment. From the assignments matched between the two datasets we see that, in line with the general pattern for all IPA projects, JASPERS’ assignments were primarily concentrated on technical assistance, but we cannot generalize this conclusion as the portfolio data in JADE does not distinguish between the these types of projects.

In terms of size of projects, based on the data from DG NEAR on project financing, we compared the median values of total costs for projects assisted by JASPERS and projects not assisted for those assignments that could be matched between the two datasets. Table 5.1.17 presents the details by type of IPA project.

Table 5.1.17 Median total amounts for IPA projects (euro)

Type IPA project	Projects not assisted	JASPERS assisted
Investment facility	13,325,000
Investment grant	7,999,200	28,375,540
Technical assistance	925,000	950,000

Note: Data covers 236 IPA projects not assisted by JASPERS, and 17 IPA projects with JASPERS support. All projects are approved and started during 2011-2019.

Source: DG NEAR data, August 2019.

On this basis we learn that, for investment grants, projects assisted by JASPERS TA are much larger, but the numbers of such projects for the two groups are very different (with only 3 for projects assisted by JASPERS advisory). Therefore this evidence is rather preliminary. As regards advisory for technical assistance, the median costs of projects between two groups (assisted and not assisted) are similar.

Stakeholder consultation

So far in this section we presented the assignments included in JASPERS’s portfolio and the corresponding projects based on data collected from databases and documents from the Commission services and JASPERS, including also comparisons with the previous programming period whenever relevant. This evidence reflects the extent and type of JASPERS’s services requested by beneficiary countries during the first 5 years of the current programming period.

In order to understand also the stakeholders’ perceptions of their experience with JASPERS services, we carried out a targeted consultation for this evaluation. The consultation was available online during the period June – September 2019, and it gathered replies from 210 respondents. Details on the composition of the pool of respondents and the distribution of replies by survey question are included in Annex 1.

The respondents to the consultation come with diverse perspectives on JASPERS’s services as they include Managing Authorities, beneficiaries or promoters of projects, professionals from consultancy companies, and staff from the European Commission. Many of these stakeholders declared that they had experience with all JASPERS’ services, and especially with technical support for preparation of major projects (87%).

Across mandates, although more than 80% of all respondents said they had experience with JASPERS services for the ESIF mandate, we also find that one third of them also declared that they experienced JASPERS’ services for more than one of the three mandates (ESIF, CEF, IPA). Therefore, as it is very likely that the opinions elicited through the consultation are informed by experiences across mandates and types of services combined, we summarize the replies to the consultation at the level of all mandates, by type of respondent.

Almost all of the 210 stakeholders responding to the consultation consider JASPERS relevant for support to increase project quality and for assisting the alignment of projects with EU environmental directives. Further, a large majority perceive JASPERS as relevant for support in developing skills for cost benefit analysis.

When asked for their opinion on the relevance of JASPERS services for their needs, a large majority of these respondents perceive JASPERS as being relevant or very relevant for helping increase projects quality (97% of the valid replies), for assisting with compliance with EU environmental directives (90%), and for developing skills for cost benefit analysis (88%). Other needs such as speeding up the approval process, increasing capacity to prepare projects and reviewing applications were also chosen by a large majority of respondents. Less frequent needs relevant for JASPERS’s services are perceived to refer to compliance with EU public procurement and compliance with EU state aid. Table A1.7 in Annex 1 includes the details for this question.

The most frequent reasons for requesting JASPERS’ services are perceptions of increased chances for faster project approval.

In terms of reasons for requesting JASPERS services, most frequent choices were increased chances for approval of projects in terms of a positive decision and speed of the decision process. Lack of skills for project preparation or for the application process was also chosen as a reason for asking for JASPERS’ assistance, albeit less frequently. Table A1.6 in Annex 1 includes the details for this question.

A large majority of stakeholders think that JASPERS has a high ability to respond to needs for improving project quality, for ensuring compliance with EU environmental regulations, and for assisting with the development of skills for cost benefit analysis.

Respondents were also asked about their perceptions of the extent to which JASPERS could respond to specific needs with its services. Overall, a very large majority of replies (85-94%) indicated a high ability to respond to needs for improving project quality, for ensuring compliance with EU environmental regulations, for helping preparing projects, and for assisting the development of skills for cost benefit analysis. Less frequent replies referred to compliance with EU public procurement and assistance for project implementation. Table A1.8 in Annex 1 includes the details for this question.

Overall, a very large majority of stakeholders rated highly JASPERS’ services in terms of coherence with EU standards, quality of advice, effectiveness and timeliness.

Finally, in terms of customer satisfaction, a very high proportion of stakeholders (84-94%) rated highly JASPERS’ services in terms of coherence of their assistance with EU standards, quality of advice, expertise and deliverables, achievement of the assignment objectives, and efficiency and timeliness of service. Table A1.9 in Annex 1 includes the details for this question.

5.2EFFECTIVENESS

The main objective of Commission’s initiative with technical assistance from JASPERS is to support Member States in preparing investment projects at EU standards and compliant with EU regulatory requirements. The advisory function aims at ensuring that projects of good quality are proposed and succeed in a timely manner with the application for EU funding so that they reach the stage of implementation as quickly as possible. For the review function, JASPERS is to provide an independent expert assessment of major projects submitted for EU financing. And for the function of horizontal advice and capacity building, the objectives include assisting in the preparation of strategies and national plans for investments supported by EU funds, helping identify viable project pipelines for the operational programmes, and enabling strengthened national technical capacity for project preparation and implementation.

In this section we analyse the extent to which and how these objectives are met by type of mandate and by type of JASPERS’ service. We analyse the effectiveness of JASPERS’ services with respect to the extent to which the projects assisted by JASPERS advisory are subsequently submitted for approval for EU financing, the quality of these projects and their success in the approval process, the effectiveness of JASPERS advisory and review functions in signalling quality issues identified later during the approval process, the chances of projects not being interrupted if they had been assisted by JASPERS during project preparation, and the correlation between JASPERS technical assistance and projects’ progress in implementation. For the review function, we also analyse the time needed for JASPERS’ assessment in overall context of the two procedures for the assessment and approval of major projects in the current period.

This section is organised by mandate and type of JASPERS’ service, and it concludes with the online stakeholder consultation on the effectiveness of JASPERS’ services. The main findings are summarised in the Text Box 5.2.1 below.

Text Box 5.2.1 Main findings for the effectiveness of JASPERS’ services 2014-2018

ESIF MANDATE

1)Major projects: Of all advisory assignments for major projects completed, 70% resulted in major projects adopted by the Commission by October 2019.

2)Major projects: Compared with the first five years of 2007-2013, JASPERS completed fewer advisory assignments for major projects in the current period, but it helped mobilize 22% more EU resources with the projects assisted and approved.

3)Major projects: The median duration of JASPERS’ advisory involvement in project lifecycle is of 20 months, although there is a high variability across assignments by sector.

4)Major projects: In 42% of cases, JASPERS’ assistance is requested during the stage of core project preparation, with a resulting median coverage of 16% of this stage by advisory in calendar terms. For 18% of assignments, JASPERS is involved from the very beginning of the core project preparation, with a full coverage of this stage in calendar terms.

5)Major projects: Involvement of JASPERS’ advisory support from the start of project preparation is positively correlated with a shorter duration of the preparation stage relative to the duration of the project lifecycle.

6)Major projects: Relative to the previous period, there has been a significant reduction from 82% to 46% in the overall rate of interruptions of major projects during the approval procedure based on Art 102.2.

7)Major projects: The two most frequent topics for EC observations in interruption letters include cost benefit analysis and feasibility study, with higher frequencies in non-assisted relative to assisted projects.

8)Major projects: Relative to assisted projects, the projects not assisted by JASPERS advisory tend to have a higher number of critical issues identified in the PSA reports by JASPERS IQR. Nevertheless, there are no significant differences between the two groups of projects in terms of the number of observations raised by the Commission in interruption letters and the duration of interruptions.

9)Major projects: The effectiveness of JASPERS’ advisory to signal issues with project quality later identified during the approval process varies across topics. It is highest for cost benefit analysis and feasibility study, but much lower for other project features such as indicators, project description, or project timetable.

10)Major projects: JASPERS’ advisory for project preparation has a positive effect on the likelihood that projects are not interrupted during the approval process based on Art 102.2. This effect is statistically significant when accounting for project features and country specific factors.

11)Major projects: JASPERS’ advisory support is positively correlated with progress in the financial implementation of major projects, but only for projects in which JASPERS intervenes early during the period of core project preparation and for younger, more recent new projects prepared.

12)Non-major projects: Overall effectiveness of JASPERS’ advisory for non-major projects could not be established as data on projects’ progress with approval and implementation is not available at portfolio level. Preliminary evidence for a third of completed assignments, however, indicates an approval rate for such projects by the Managing Authorities of at most 60%.

13)Non-major projects: The median duration of an advisory assignment for non-major projects is 14 months, although there is significant variability across assignments.

14)Non-major projects: Most frequent issues raised by JASPERS advisory during project preparation relate to procurement and implementation, cost benefit analysis, and risk and sensitivity analysis.

15)Review function: The median total duration of the process of assessment and approval of major projects notified for Art 102.1 is 7 months. Of this, the IQR assessment carried out by JASPERS IQR, including also the response time for the Member States, has a median duration of 5 months.

16)Review function: The median total duration of the process of assessment and approval of major projects submitted for adoption based on Art 102.2 is 3 months for projects not interrupted, and 6 months for projects interrupted. In this process, JASPERS IQR is highly effective in delivering timely PSA reports (within one month).

17)Review function: Relative to the period 2007-2013, there has been a significant reduction in the overall time taken with the assessment and approval of major projects. This reduction is highest for the procedure based on Art 102.2 (with a reduction of 45%) and lower for the procedure based on Art 102.1 (with a 12% reduction).

18)Review function: The effectiveness of JASPERS IQR to signal issues of project quality in the PSA report is relatively high for most topics, and especially for issues related to project timetable, risk and sensivity analysis, feasibility study, and cost and benefit analysis.

19)Capacity building and HSS: So far, while a large majority activities for capacity building were completed and achieved their objectives for delivering trainings for the Member States, such activities have not proven sufficient to trigger a visible ripple effect on the technical capacity of Member States to address effectively issues such as cost benefit analysis, climate change and environment in project preparation.

20)Capacity building and HSS: JASPERS has proven potential to promote more strategic and innovative approaches for building technical capacity, but such approaches were not yet mainstreamed at the portfolio level during the reference period.

21)Capacity building and HSS: For the horizontal and strategic support, based on an analysis of sampled assignments, there are indications that JASPERS’ support helps compensate for lower local technical capacity for guiding and supervising tasks on specialised topics of policy relevance.

CEF1.1 MANDATE

22)CEF1.1: For around 70% of the CEF assignments for works completed until end 2018, JASPERS’ technical assistance was requested primarily around the time of project applications. Some of these projects, however, had been assisted by JASPERS for project preparation during the previous programming period (for the ESIF mandate).

23)CEF1.1: In 40% of cases, the studies financed by CEF and assisted by JASPERS advisory based on the CEF1 pilot mandate were feasibility studies prepared for investment projects intended for the CEF financial support.

IPA MANDATE

24)IPA: More than 70% of the assignments included in JASPERS’ IPA portfolio are for technical advisory work. So far, close to 20% of these assignments resulted in IPA projects approved for EU-funding due to a combination of factors external to JASPERS’ services.

25)IPA: Based on an analysis of sampled assignments, there is indication that the effectiveness of JASPERS’ advisory support is limited in cases when the requests for support arrive late in the project preparation cycle.

26)IPA: Beneficiaries appreciate JASPERS’ horizontal and strategic support especially as it helps raise awareness and familiarize national staff with respect to the management and appraisal of EU funded projects.

ONLINE CONSULTATION

27)More than 80% of the stakeholders responding to the consultation considered that JASPERS had a significant, and even decisive, effect on project technical specifications and its documentation. They also rated highly JASPERS’ assistance for aligning projects with EU and national policies, and for enabling faster project preparation and approval.

For each of these evaluation findings the supporting evidence is as follows.

ESIF Mandate

Technical advisory for major projects

Of all advisory assignments for major projects completed, 70% resulted in major projects adopted by the Commission by October 2019.

As presented earlier, for the ESIF mandate, a significant proportion of JASPERS’ work is dedicated to providing technical assistance for major projects with the objective to assist Member States in getting these projects ready to apply for EU funds. Over the period 2014-2018, JASPERS’ portfolio of assignments for technical assistance included 414 assignments for major projects, with 178 of these assignments completed during 2018 and additional 48 completed during 2019. Of the completed assignments, 158 (70%) of them resulted in major projects adopted by the Commission by October 2019 123 . The adopted projects assisted by JASPERS cumulate a total EU funding of 23.9 billion euro (ERDF and Cohesion Fund combined), representing 65% of EU resources mobilised by all major projects adopted by the Commission during the period analysed. Three quarters of the projects assisted by JASPERS are new major projects, while the remaining projects are phased from 2007-2013.

Compared with the first five years of 2007-2013, JASPERS completed fewer advisory assignments for major projects in the current period, but it helped mobilise 22% more EU resources with the projects assisted by advisory and approved by the Commission.

By comparison, during the previous programming period until 2011, JASPERS completed 343 assignments for major projects, resulting in 205 major projects adopted by the Commission by October 2012. The adopted projects assisted by JASPERS cumulated a total of 19.6 billion euro (ERDF and Cohesion Fund combined), representing 65% of EU resources mobilised by all projects adopted over the first five years of the programming period.

Therefore, when comparing the two periods, we see that in the current period JASPERS advisory completed fewer assignments for major projects by the 5th year of the programming period, resulting also in fewer projects adopted by the Commission. Nevertheless, In terms of EU financing, while their relative contribution of 65% remains the same over the two periods, the projects assisted by JASPERS and adopted by the Commission in the current period tend to be larger in size, and therefore they mobilise more EU resources (by 4 billion euro) even if fewer in number.

The median duration of JASPERS’ advisory involvement in project lifecycle is of 20 months, although there is a high variability across assignments by sector.

In terms of duration of JASPERS’s assignments for technical assistance, we find that the median duration of completed assignments is 20 months (and average 27 months), compared to an average of 16 months in period 2007-2013. We also find that there is significant variability across assignments in this regard. In Table 5.2.1 we summarise the data on duration of completed assignments for technical assistance for major projects by sector.

Table 5.2.1 Duration of advisory assignments for major projects (months)

Sector	Number assignments	Median duration	Minimum duration	Maximum duration
Energy and solid waste	23	19	2	60
Rail, air and maritime	72	19	0.2	76
Roads	36	17	1	129
Smart development and ICT	23	15	1	95
Water and wastewater	24	49	8	94
All advisory assignments	178	20	0.2	129

Note: Data covers all assignments completed by December 2018

Source: JASPERS JADE database, December 2018

At sector level, we learn that, while for most of the sectors assisted by JASPERS the median duration is between 15 and 20 months, the assignments for the sector of Water and wastewater tend to take much longer. High variability across assignments for all sectors, however, is indicated by the fact that duration across assignments can vary from very short (a minimum of 6 days for Rail, air, and maritime) to very long (a maximum of 129 months for Roads) across sectors.

In order to understand better the role of JASPERS advisory in the process of project preparation we compared the duration of assignments with the duration of project preparation in Member States. For this purpose, we collected SFC2014 data on dates for project preparation and for the overall (planned) duration of all new major projects submitted to the Commission until October 2019. This data is summarised in Table 5.2.2.

Table 5.2.2 Lifecycle of new major projects from preparation to completion (in months)

Sector	Number projects	Median project preparation	Median project duration	Median ratio: Preparation / Total duration
Energy and solid waste	16	44	89	50%
Rail, air and maritime	67	59	101	58%
Roads	76	135	166	85%
Smart development and ICT	25	40	67	60%
Water and wastewater	26	64	117	53%
All new projects submitted	210	82	125	69%

Note: Data covers 210 (assisted and non-assisted) new major projects submitted to SFC2014, based on project timetables.

Source: European Commission, SFC2014, October 2019

Durations reported in Table 5.2.2 are calculated based on the project timetables reported in the application to the Commission. These timetables include the start and end dates for 10 stages of project development 124 . For the calculation of overall duration of a project we considered the earliest and latest dates reported in the project timetable, while for the preparatory phase of the project we considered the earliest and latest dates covering all core stages for preparation (feasibility study, cost benefit analysis, environmental impact assessment and design studies) up to, but not including, tender documentation.

On this basis we find that, from cradle to completion, major projects can last a very long time, with longest total durations for Roads (14 years median value). The data also shows that the preparation of projects has different relative weights in total duration across sectors. For Roads, for example, the project preparation takes around 85% of the overall project duration, while for Rail, air, maritime and for Smart development and ICT the preparatory phase accounts for up to 60% of the overall duration of the project.

In 42% of cases, JASPERS’ assistance is requested during the stage of core project preparation, with a resulting median coverage of 16% of this stage by advisory in calendar terms. For 18% of assignments, JASPERS is involved from the very beginning of the core project preparation, with a full coverage of this stage in calendar terms.

In this context, it may seem that the time taken by JASPERS’ assignment for assistance with preparation of a project is about a quarter of the total preparation time. This, however, may not be the case in situations where JASPERS’ support is requested later during the preparation phase, or when the assignment extends also after the core preparation is completed. For this reason, we also calculated the overlap between period of JASPERS’s assignment and the period for the project preparation as follows.

As explained in the section for relevance, for the new major projects assisted by JASPERS advisory and submitted to the Commission we used the data on project timetables and the calendar duration of assignments to identify three groups of assignments: Group 1: assignments created from the start of the preparatory stage of the project or earlier; Group 2: assignments created after the preparatory stage has started; and Group 3: assignments created after the preparatory stage ended. For the first two groups we calculated the overlap in % terms between the duration of the assignment and the duration of the preparatory stage as defined above. This gives the coverage of the preparatory stage by the assignment, with higher values indicating a higher coverage of core project preparation by advisory in terms of (calendar) duration. This data is reported in Table 5.2.3.

Table 5.2.3 Overlap between advisory assignment and project preparation

Group assignments	Number assignments	Coverage preparatory by assignment
		Median	Max	Min
Group 1 (from start)	21	99%	100%	34%
Group 2 (after start)	89	16%	95%	0.10%
Group 3 (after completion)	10	0	0	0

Note: Data covers 120 assignments for new major projects assisted by JASPERS and submitted to the Commission 125 .

Source: European Commission SFC2014, October 2019; JASPERS’ JADE database, December 2018

This data indicates that 89 of these assignments were created after the start of the project preparation and, for a majority of these projects, they covered around 16% of the preparation time. Data also indicates that there is significant variability across these assignments, as some could last for almost the full period of the core preparatory stage. Further, we also found a group of 21 assignments which started from the very beginning of the preparatory stage (or earlier) and a majority of them lasted for full core preparation. Finally, 10 of the assignments analysed had no overlap as they started after the core preparatory stage had been completed. Overall, we could infer that around a quarter of the assignments analysed were created at early or very early stages of the preparatory phase for projects, possibly with higher chances to advise on critical features of project design.

Involvement of JASPERS’ advisory support from the start of project preparation is positively correlated with a shorter duration of the preparation stage relative to the duration of the project lifecycle.

Next we also analysed the correlation between the duration of the stage of core preparation (relative to total duration of project’s lifecycle) and the stage of involvement of JASPERS’ advisory in this process. The duration of project preparation can be a function of several co-founding factors such as projects features (size and sector) and country specific effects. For this reason, we carried out linear regression models for the ratio of the duration of core preparation in total project lifecycle for which we tested two main sets of variables for JASPERS’ assistance: 1) JASPERS’ advisory assistance at any stage in project preparation relative to support after core preparation or no assistance, and 2) JASPERS’ involvement in project preparation from start and after start of core preparation, relative to involvement after core preparation or not at all. The analysis covers 79% of all new major projects submitted to the Commission until October 2019 126 . Table A2.2.4 in Annex 2 includes the results.

On this basis, we learn that JASPERS’ advisory support is positively correlated with a shorter duration of core preparation relative to project lifecycle only when the support is requested from the start of the preparation stage. This effect is statistically significant when controlling for project features (size and sector) and country fixed effects. Moreover, we also find some indication that later involvement of JASPERS’ advisory in project preparation is slightly correlated with a longer duration of project preparation. This effect, however, is statistically significant only at 10% level, and therefore weaker.

Relative to the previous period, there has been a significant reduction (from 82% to 46%) in the rate of interruptions of major projects during the approval procedures.

Next, we look at what happened after advisory was completed and at the extent to which projects assisted by JASPERS advisory were sufficiently prepared for the adoption / approval in the Commission, in comparison with projects not assisted by JASPERS. This analysis covers all 150 new major projects submitted to the Commission for Art 102.2 during the reference period (whether assisted or not), with a focus on the projects interrupted in the approval process in the Commission 127 . Data for assisted and not assisted projects and rates of interruptions by sector is presented in Table 5.2.4.

Table 5.2.4 New major projects interrupted by the Commission by sector and support

Sector	Assisted by JASPERS advisory		Not assisted by JASPERS advisory
	Number projects	% interrupted	Number projects	% interrupted
Energy and solid waste	11	55%	2	100%
Rail, air and maritime	34	38%	6	83%
Roads	21	52%	44	30%
Smart development and ICT	12	50%	6	83%
Water and wastewater	14	57%	0	n.a.
Total	92	48%	58	43%

Note: data covers all 150 new major projects submitted for Art 102.2.

Source: European Commission SFC2014, October 2019.

Overall, we find that 69 of the 150 new major projects submitted by Member States for EU financing were returned for modification or, to a much lesser extent, withdrawn by the Member State for quality issues. This amounts to an overall rate of interruptions of 46%. Further, If we consider separately the two groups of projects (assisted and not assisted by JASPERS advisory), we note that the rate of interruption for projects assisted by JASPERS advisory was 48%, and it was 43% for the non-assisted projects. As we shall see later, however, the higher rate of interruption for projects assisted by JASPERS is not attributable to JASPERS advisory but rather to project features and country specific factors.

By comparison, based on an analysis of 208 major projects submitted (of which 168 assisted by JASPERS advisory) for 2007-2013, we learn that the overall rate of interruption was much higher (82% of all projects submitted), with the similar rates of interruption (82%) for the two groups: assisted and not assisted. Therefore, relative to the previous period, there has been a significant reduction in the rates of interruption of major projects. Nevertheless, given that the extent of this reduction appears to be roughly in a comparable range for the two groups of projects (assisted and not assisted), we interpret this improvement between the two periods as primarily due to the overall changes in the regulatory and administrative set-up for the assessment and approval of projects instituted by the Commission for 2014-2020 128 .

As regards the distribution by sector in the current period, we note that, compared with the non-assisted ones, projects assisted by JASPERS advisory tend to have lower rates of interruption for all sectors, except for Roads. For the sector of Water and wastewater, all projects submitted had been assisted by JASPERS advisory, and therefore the comparison is not feasible. Finally, within the group of assisted projects across sectors, we learn that the rate of interruption tends to be higher for Energy and solid waste and for Water and wastewater.

Finally, when comparing rates of interruption for assisted and not assisted projects at country level, we find that the projects assisted by JASPERS advisory tend to have zero or lower rates of interruption for 5 countries, similar rates of interruption for 3 countries, and a higher rate for only one country. Table A4.6 in Annex 4 includes the details.

The two most frequent topics for EC observations in interruption letters include cost benefit analysis and feasibility study, with higher frequencies in non-assisted relative to assisted projects.

In order to understand better why projects are still interrupted when submitted to the Commission, we analysed also the reasons for interruptions. For this analysis, we collected structured data on Commission’s observations made in the observation letters by topic for all 69 projects interrupted and/or withdrawn until Oct 2019. This data is presented in Table 5.2.5.

Table 5.2.5 Topics of EC observations for major projects submitted for Art 102.2

Topic EC observations	% in all for Art. 102_2	Interrupted projects			% in total interrupted
		Frequency in not assisted	Frequency in assisted	Number projects
(1)	(2)	(3)	(4)	(5)	(6)
Cost-Benefit analysis	31%	72%	66%	47	68%
Feasibility study	27%	72%	50%	40	58%
Environmental issues	25%	60%	52%	38	55%
Project description	24%	44%	57%	36	52%
Total eligible costs	23%	48%	52%	35	51%
Risk and Sensitivity analysis	21%	48%	43%	31	45%
Financial plan	19%	32%	45%	28	41%
Indicators	18%	28%	45%	27	39%
Capacity beneficiary	9%	16%	20%	13	19%
Project timetable	8%	8%	23%	12	17%
State aid	7%	12%	18%	11	16%
Project not in OP	1%	4%	2%	2	3%
Not a major project	1%	4%	0%	1	1%

Note: Data covers all 69 projects interrupted, of which 44 assisted by JASPERS.

Source: European Commission’s interruption letters and SFC2014 for projects submitted until October 2019.

Table 5.2.5 reads as follows. Column (2) includes the share of projects with Commission’s observations for the respective topic in all projects submitted for Art. 102.2 during the period analysed. For Cost-Benefit analysis, for instance, 31% of the projects submitted to the Commission for Art. 102.2 had observations related to this topic. Column (6), on the other hand, refers only to project interrupted and gives the share of projects with observations on the respective topic in all projects interrupted. Therefore, of all 69 projects interrupted, 68% had observations related to Cost-Benefit Analysis. Finally, the intermediate columns (3) and (4) indicate the frequency of observations related to a given topin in the assisted, respectively non-assisted, projects.

On this basis, we learn that around a third of projects submitted to the Commission for Art. 102.2 were interrupted and had observations related to cost-benefit analysis and feasibility study in the interruption letters (column 2 in Table 5.2.5). Other frequent reasons for Commission’s observations include also environmental issues, project description, total eligible costs, risk and sensitivity analysis (all above 20%) and, to a lesser extent, the financial plan and indicators in the project. The frequency of issues for cost benefit analysis, feasibility study and environmental issues tends to be rather high within the group of interrupted projects (as indicated in column 6).

When we compare topics of interruption for assisted and not assisted projects (columns 3 and 4 in Table 5.2.5) we learn that for three of these topics (cost benefit analysis, feasibility studies and environmental issues) the frequency of their presence in observation letters tends to be higher for the non-assisted projects. For the remaining topics, however, the frequency varies between assisted and not assisted, with several topics (project description, total eligible cost, financial plan etc.) being more frequent for the assisted projects. This may also be due to the fact that these issues are identified in the project applications, which sometimes are submitted by Member States with incomplete information.

By comparison, for period 2007-2013, highest rates of interruptions were related to environmental issues, financial plan, and project design (all above 30% in projects submitted and analysed). Table A3.9 in Annex 3 presents these details. When comparing the rates by topic for the two periods, we note reductions in the frequency of Commission’s observations especially for topics related to the financial plan, environmental issues, and implementation timetable and structures, thus indicating a learning process in the Member States in this regard between the two periods. This comparison, however, is rather inferential as the data for the two periods is not perfectly comparable due to slightly different classifications of topics for interruption 129 .

Relative to assisted projects, the projects not assisted by JASPERS advisory tend to have a higher number of critical issues identified in the PSA reports issued by JASPERS IQR. Nevertheless, there are no significant differences between the two groups of projects in terms of the number of observations raised by the Commission in interruption letters and the duration of interruptions.

Next, for a further understanding of the maturity of projects submitted to the Commission, as proxies for the quality of projects interrupted, we also calculated the number of observation topics and the number of observations raised by the Commission in the observations letter per project. Our conjecture is that fewer topics or issues raised would indicate a higher maturity of project preparation at adoption 130 . On this basis we find that the median number of observation topics per project is 4 (out of 13), and the median number of observations raised per interrupted project is 6, with no significant differences between assisted and non-assisted projects for both medians values.

Further, we analysed this issue also based on the number of critical issues identified by JASPERS IQR in the PSA reports. On this basis, we learn that the median number of critical issues identified by JASPERS IQR for the non-assisted projects is 4, while the median number of critical issues identified for the projects assisted by JASPERS advisory is 1, and the difference in medians is statistically significant.

Another possible indication of the maturity of projects arriving to the Commission is also the time it takes to resolve the issues raised in the Commission’s assessment. For all project interrupted and subsequently adopted, we calculated the time difference between the date when the project was adopted and the date of first interruption (or withdrawal). For the 47 projects adopted after having been interrupted over the reference period, we find that the median total duration for interruptions is 4.4 months (with values ranging between 1 to 22 months), and that there are no significant differences in the median values for the assisted and non-assisted projects. Therefore, interrupted projects in the two groups (assisted and not assisted) tend to have similar median durations of interruption.

The effectiveness of JASPERS’ advisory to signal issues with project quality later identified during the approval process varies across topics. It is highest for cost benefit analysis and feasibility study, but much lower for other project aspects such as indicators, project description, or project timetable.

The evidence presented so far indicates that, for a third of the projects, JASPERS advisory assistance does not seem to be sufficient to ensure that the projects advised and submitted to the Commission are of sufficient quality to warrant adoption without interruption. There could be several reasons why this is observed. First, it is possible that projects are interrupted for issues which had not been under the scope of JASPERS’ advisory assignment. Second, it is also possible that even if JASPERS advised the beneficiary on specific topics, its comments were not taken on board by the time the project was submitted to the Commission. Third, it is possible that JASPERS advisory reviewed the respective topics but did not raise the issues subsequently raised by the Commission in its observation letters during the adoption procedure.

In order to distinguish among these three cases, for the projects assisted by JASPERS advisory and interrupted by the Commission, we compared the topics on which the Commission raised observations with the contribution and comments documented in the action completion notes of the corresponding advisory assignments. This analysis covers 37 major projects interrupted by the Commission during the procedure for Art 102.2 and which had been supported by JASPERS advisory. These projects represent 84% of all new major projects advised by JASPERS for project preparation and subsequently interrupted during the adoption procedure 131 .

From the ACNs issued by JASPERS advisory upon completion of an assignment we collected structured data on the type of JASPERS assistance to the project as follows. Following the structure of the ACN, we coded the assistance by topic as: 1) Significant issue, if JASPERS experts raised at least one significant issue for the respective topic, 2) Comment, if JASPERS experts included suggestions for improvements in the comments for the respective topic, 3) Contribution, if no significant issue or suggestions for improvements were found, but there was an explicit mention by the experts that they contributed to the issue during the assignment, 4) Positive assessment, when we found a positive assessment of the topic but no evidence for 1) to 3), and 5) Review, when the experts provided a description for the topic without an assessment. This data is reported in Table 5.2.6 for the projects that had observations from the Commission on the respective topic during interruptions.

Table 5.2.6 JASPER’ assessments in ACNs for interrupted projects by topic of interruption

Note: Data covers 37 projects assisted by JASPERS, interrupted by the Commission, and for which we found the ACN provided by the Member State in SFC2014 together with the project documentation.

Source: JASPERS’ ACNs, European Commission’s SFC2014 and observation letters, October 2019.

Table 5.2.6 reads as follows. For example, for cost benefit analysis, in column (7) we learn that 22 projects had observations from the Commission on issues of cost benefit analysis, and they had also been assisted by JASPERS advisory on the topic. For 8 of these 22 projects JASPERS’ experts raised significant issues with cost benefit analysis in the ACN, for 11 of them JASPERS included comments for improvements without raising significant issues, and for the remaining 3 the experts stated they contributed to the analysis of the projects.

On this basis, we calculate that JASPERS advisory had signalled issues in this regard in the ACN either by raising significant issues or by including suggestions for improvement for 86% of projects with observations on cost benefit analysis from the Commission. In terms only of significant issues raised by JASPERS advisory, however, the proportion reduces to 36%. The same reasoning applies to all remaining topics.

From this data we infer the following. First, we note that for the first 5 topics in the table (from cost benefit analysis to capacity of beneficiary) JASPERS’ experts signalled changes needed in projects for more than 57% of the projects that later received observations from the Commission on the respective topics (column 8). This suggests that, even if experts included significant issues and comments in their reports, in some cases the issues were not resolved by the Member State before submitting the project to the Commission.

Nevertheless, based on the proportion of significant issues signalled by JASPERS (column 9), we also infer that, in some cases, the experts’ message on issues which required further attention might not have been sufficiently strong for the beneficiary to understand that changes were needed. This is based on the fact that many of the projects assisted by JASPERS were interrupted for topics on which the ACNs did not raise significant issues but included comments.

And third, there are also projects which were interrupted and received observations from the Commission, but for which the ACN included a positive assessment, with no suggestions for improvement. This is the case, for instance, for the assessment of environmental issues, indicators, and project description for which two thirds of the projects had no indication in the ACN that some aspects might have needed improvements.

JASPERS’ advisory has a positive effect on the likelihood that projects are not interrupted during the approval process for Art 102.2. The effect is statistically significant when controlling for project features and country specific factors.

Finally, as regards interruptions, we also analysed the probability that a project is not interrupted as a function of the stage in which JASPERS’ support is requested for project preparation and other co-founding factors such as project size, sector and country specific factors. For JASPERS’ advisory support, we tested a variable with the following values: 1 – if JASPERS advisory assisted the project preparation from the beginning or at least during the first half of the core preparation period (early involvement), 2 – if JASPERS advisory assisted the projects in the second half of the core preparation period (later involvement), and 3 – if JASPERS assisted the project after the end of the core preparation period (late involvement). As explained earlier, core preparation period is defined as including all stages until preparation of tender documentation and it is calculated based on project timetables and period of assignment. The ordered logistic regression models run for this analysis are presented in Table A2.2.6 in Annex 2.

Covering 95% of all new major projects submitted to the Commission for Art 102.2 until October 2019, this analysis indicates that major projects are more likely not to be interrupted during the approval / adoption process if they had been advised by JASPERS advisory during the preparation of the projects. This effect is higher and more statistically significant when JASPERS provides assistance from the beginning of the preparation period or at least during its first half, when compared with later involvement in project preparation.

JASPERS’ advisory support is positively correlated with progress in the financial implementation of major projects, but only for projects in which JASPERS intervenes early during the period of core project preparation and for younger, most recent new projects prepared.

Next, we assessed the effectiveness of the advisory function also in relation with the progress in project implementation after its adoption/ approval. For this purpose we collected the data on financial and physical progress for 166 new major projects submitted to the Commission 132 , as reported by Member States in the annual implementation reports for 2019 (or for 2018, if 2019 missing). First, the data on physical implementation is reported in Tables 5.2.7.

Table 5.2.7 Physical implementation of new major projects adopted

Progress implementation (reported by Managing Authority)	JASPERS TA assisted		Not assisted
	Number	% in total	Number	% in total
Advanced construction	27	25%	27	47%
Completed/ In operation	13	12%	12	21%
Construction	61	56%	17	30%
Design	3	3%	1	2%
Procurement	5	5%	0	0%
Total	109	100%	57	100%

Note: Data covers 166 projects submitted to the Commission and with implementation data.

Source: European Commission, SFC2014, AIRs 2018-2019, downloaded September 2020.

The large majority (81%) of projects assisted by JASPERS advisory were in construction or advanced construction by end 2019, and 12% were already completed and/or in operation. By comparison, for the non-assisted projects, 77% were in construction or advanced construction, and 21% completed. A minority were still in design and procurement phases for both groups.

The financial progress in implementation until end 2019 is based on the percentage of certified eligible expenditure declared by Member States at project level, and it is reported in Table 5.2.8.

Table 5.2.8 Financial implementation of new major projects adopted

Financial progress (% certified expenditure)	Assisted		Not assisted
	Number	% in total	Number	% in total
1_No progress 0%	20	18%	7	12%
2_Progress up to 10%	18	17%	2	4%
3_Progress in 10% -50%	37	34%	10	18%
4_Progress in 50%-80%	17	16%	6	11%
5_Progress in 80%-100%	13	12%	20	35%
6_Progress more than 100%	4	4%	12	21%
Total	109	100%	57	100%

Note: Data covers 166 projects submitted to the Commission and with implementation data.

Source: European Commission, SFC2014, AIRs 2018-2019, downloaded September 2020

In terms of certified eligible expenditure, 32% of projects assisted by JASPERS advisory had certified at least 50% of the eligible expenditure of the project by end 2019, compared with 67% of the non-assisted projects. This indicates that, over the reference period, the non-assisted projects are relatively more advanced with financial implementation relative to assisted projects.

Progress in implementation, however, can be the result of several co-founding factors such as project size, sector, year of start of implementation, and country specific factors. For this reason, for the same sample of 166 projects, we analysed whether JASPERS advisory had an effect on the likelihood of higher financial and/or physical progress in implementation while controlling for projects features and country fixed effects. For JASPERS advisory we tested three variables: 1) JASPERS support at any stage during core preparation of projects, 2) JASPERS support at early stages of core preparation of projects, and 3) JASPERS support at early stages of core preparation of young new projects, where young projects are defined as projects which started implementation at earliest in 2017. The method used is ordered logistic regression and the results for the models of financial progress in implementation are presented in Table A2.2.8 in Annex 2.

Based on this analysis we find that there is a positive correlation of JASPERS advisory with the financial implementation of new major projects adopted only when JASPERS assistance is requested during early stages of project preparation, and only for young projects. We do not find such an effect for JASPERS assistance in general and not even only for early assistance without accounting for the projects’ start of implementation. One reason for this result is the fact that 71% of the assignments with early involvement of JASPERS are for young projects starting implementation by 2017. For the other two groups (i.e. with later involvement of JASPERS in preparation or with no involvement at all), the frequency of such projects is much lower (at most 35%). Overall we find that, for almost a majority (47%) of the more recent new projects analysed, Member States requested JASPERS advisory at early stages of preparation and this is positively correlated with the subsequent financial implementation of these projects after approval for EU funds.

As regards physical implementation, the effect of JASPERS advisory is not statistically significant but we believe that this is due to the fact the classification of implementation stages self-reported by the Member States is not sufficiently granular to enable a similar analysis by stage of JASPERS advisory and effective maturity of physical implementation 133 .

Therefore, on this basis, we conclude that the evidence collected indicates that, in addition to being correlated with a higher probability of projects not being interrupted by the Commission (therefore indicating better project quality), JASPERS advisory is also positively correlated with the financial implementation of the assisted projects if the support was requested early during project preparation, and for younger projects.

Technical advisory for non-major projects

Overall effectiveness of JASPERS’ advisory for non-major projects cannot be established as data on projects’ progress with approval and implementation is not available. Preliminary evidence for a third of completed assignments, however, indicates an approval rate of at most 60%.

The possibility to evaluate the effectiveness of JASPERS’ advisory work for non-major projects is much more limited due to lack of data. As JASPERS does not collect any data on what happens with the projects once the advisory work is completed, we requested this information from 6 Member States sampled for the evaluation, but only two of them responded. Therefore, except for the data on duration of assignments which covers the full portfolio, all other data reported in this sub-section should be interpreted as rather illustrative.

The 2014-2020 portfolio includes 140 advisory assignments for non-major projects, of which 47% were completed by December 2018. For the completed assignments, we could collect data for 74% them, i.e. from 49 JASPERS’ ACNs issued upon completion of the advisory work. On this basis, we learnt that these projects have a cumulated total cost of close to 2 billion euro, but we cannot tell to what extent these resources were mobilised through the programmes as we do not know which of these projects were eventually approved by Managing Authorities, and with which costs. For example, from the first country which responded to our request for information, we learnt that 58% of the non-major projects with advisory work completed by December 2018 were approved by the Managing Authorities until autumn 2019. For the second country which responded to our request, one out of two projects with completed advisory was approved. These two countries together, however, represent only 30% of the completed assignments for non-major projects.

By comparison, in the period 2007-2013, the number of advisory assignments completed by JASPERS during the first five years of the period was 101, representing 76% of the 133 advisory assignments completed for non-major projects over the whole period. Also for the previous period, however, we do not know how many of these projects were eventually implemented in the operational programmes for the period.

The median duration of an advisory assignment for non-major projects is 14 months, although there is significant variability across assignments.

As regards the duration of advisory assignments for non-major projects, based on data from JASPERS’ database, we learn that the median duration across all assignments is 14 months, with higher durations for Roads and, to a lesser extent, for Water and wastewater. Table 5.2.9 presents the details.

Table 5.2.9 Duration advisory assignments for non-major projects (in months)

Sector	Number assignments	Duration assignment non-major projects
		Median	Max	Min
Energy and solid waste	8	13	45	4
Rail, air and maritime	5	13	26	4
Roads	3	41	59	5
Smart development and ICT	22	10	58	1
Water and wastewater	28	19	72	4
All completed assignments	66	14	72	1

Note: data covers all completed assignments

Source: JASPERS’ JADE database, December 2018

Nevertheless, this data also indicates that there is a significant variability across assignments, with duration ranging from one month to 72 months. This is consistent with the earlier finding that, depending on the scope of the assignment, JASPERS’ support is requested by Member States at various stages during project preparation.

For the overall duration of project preparation, however, we could only present illustrative examples as relevant data across all assignments is not available. Based on our review of the 49 ACNs for completed assignments for non-major projects, we found data on projects’ timetables only for 6 of them. As for major projects earlier, also for these non-major projects we calculated the duration of core project preparation and classified them by the stage at which JASPERS advisory is requested. We found that the median duration of core project preparation for these 6 projects ranged between 13 and 99 months, and for 4 of them JASPERS’ assistance started from the beginning of project preparation, or even earlier. For the remaining two projects, JASPERS advisory started during project preparation.

Further, we calculated also the overlap in calendar time between JASPERS’ advisory assignment and core project preparation. For the 4 assignments with early support, the record in JADE indicates that all assignments lasted for the full duration of the core preparation of the projects, extending also to the subsequent stages for preparation of tenders and for construction 134 . For the remaining two assignments, for which JASPERS’ assistance was requested after the start of project preparation, the median overlap in calendar time with project preparation was 33%.

Most frequent issues raised by JASPERS advisory during project preparation relate to procurement and implementation, cost benefit analysis, and risk and sensitivity analysis.

In terms of the content of JASPERS’ assistance for non-major projects, as with the major projects, we collected structured data on the type of contribution reflected in the ACNs of the 49 completed assignments analysed. Data collection followed the same method of classifying JASPERS assistance by topic into 5 groups explained earlier: 1) significant issues raised in the ACN, 2) comments with suggestions for improvement, 3) contribution mentioned by JASPERS experts in ACN, 4) positive assessment when 1) to 3) do not apply, and 5) review of the topic with no explicit assessment and when 1) to 4) do not apply. This data is summarised in Table 5.2.10.

Table 5.2.10 JASPERS advisory assessment in ACNs by topic of interruption

Note: Data covers 49 non-major projects assisted by JASPERS

Source: JASPERS’ ACNs

From this analysis we learn that JASPERS advisory made comments with suggestions for improvement and, to a much lesser extent, raised significant issues for almost half of the non-major projects analysed for issues related to procurement and implementation. For cost benefit analysis, risk and sensitivity analysis and demand analysis, comments were included for around 40% of the projects. For the remaining topics, the extent of contribution varies, also due to the fact that the scope of these assignments was more limited for those projects where JASPERS advisory issued a short ACN (as explained earlier in the section on relevance).

Nevertheless, when compared with the analysis for major projects, for the advisory for non-major projects the evidence on effectiveness cannot be conclusive as no sufficiently representative data is available on the success of these projects in the process of approval and, if applicable, implementation.

Review function

The newly introduced JASPERS’ review function for the ESIF mandate has the objective to ensure consistency in the assessment of major projects submitted for EU financing and reduce the time of assessment and approval of these projects.

As explain earlier, there are two ways in which the assessment of major projects takes place : 1) a complete assessment carried out directly by JASPERS IQR (the procedure based on Art 102.1), and 2) the assessment carried out by the Commission with the assistance of JASPERS IQR through the PSA service (the procedure based on Art 102.2). As regards the approval/adoption of major projects, the procedure based on Art 102.1 requires Member States to notify the Commission of a new major project by submitting the project application and an IQR report from JASPER IQR confirming that the project is of sufficient quality for approval. For the procedure based on Art 102.2, the Member State submits the full project documentation to the Commission, the Commission assesses the quality of the project and, if it considers that the project is sufficiently mature, the Commission issues a decision to adopt the project. If the quality is not considered sufficient, the project is interrupted.

For both these procedures, we calculated the time it takes to assess and subsequently approve a project and considered the contribution of JASPERS IQR in the process as follows.

The median total duration of the process of assessment and approval of major projects notified for Art 102.1 is 7 months. Of this, the IQR assessment carried out by JASPERS IQR, including also the response for the Member States, has a median duration of 5 months.

For the procedure based on Art 102.1, we calculated the assessment time by considering the difference between the date when the IQR assessment was requested by the Member State and the date when the final IQR was issued by JASPERS IQR. The approval time for these projects is calculated as the calendar time between the date when the project application and the IQR report are notified to the Commission and the date when the project is approved. Table 5.2.11 below summarizes the results by sector.

Based on the analysis of 50 major projects approved based on Art 102.1, we find that the median duration of IQR assessment carried out by JASPERS IQR division is of 5 months across all projects, varying between 1 and 15 months at project level 135 . This duration includes also the time needed by the Member State to reply to the first assessment sent by Jaspers, and therefore it represents the time elapsed from the submission of IQR until the IQR report is issued.

Table 5.2.11 Assessment and approval time for major projects – Art 102.1 (IQR) (in months)

Source : JASPERS’ JADE database (December 2018) and European Commission SFC2014 (October 2019)

At sector level, we find that the median duration for the IQR assessment is longer for Rail, air and maritime and for Roads (6 months for both).

Further in the process, once a JASPERS IQR issues a positive IQR assessment and the project is notified to the Commission, then it takes 3 months for its approval 136 .

We also calculated the overall duration of this process from the request for the IQR assessment to JASPERS IQR and until the project approval in the Commission (thus including also the time taken by Member State to submit the project application after IQR) and found that it took around 7 months overall, with a mininum of 4 months and a maximum of 18 months. As shown already, longer durations in this process are primarily driven by the duration of the IQR assessment, as the Commission takes at most 3 months for approval for all projects submitted for Art 102.1.

Furthermore, when testing the duration of the IQR assessment for projects assisted by JASPERS advisory relative to projects not assisted, we find that the median duration is higher for projects not assisted by JASPERS advisory (6 months) relative to projects assisted (4 months).

The median total duration of the process of assessment and approval of major projects submitted for Art 102.2 is 3 months for projects not interrupted, and 6 months for projects interrupted. In this process, JASPERS IQR is highly effective in delivering timely PSA reports.

For the procedure based on Art 102.2, as assessment and approval are carried out by the Commission in one procedure, we calculated the calendar time it takes for the overall process, from the date when the Member State submits the project documentation to the Commission for the first time and until the date when the project is adopted. This calendar time includes also the interruption periods for projects found of insufficient quality by the Commission in its assessment. This data is summarised in Table 5.2.12.

Table 5.2.12 Duration for procedure Art 102.2 until adoption (in months)

Projects by interruption	Number projects	Duration Art 102.2
		Median	Max	Min
Projects not interrupted	78	3	4	1
Projects interrupted	47	6	23	1.3
All projects for Art. 102.2	125	3.3	23	1

Source : JASPERS’ JADE (December 2018) and European Commission SFC2014 (Oct 2019)

For this procedure, we find that projects which are not interrupted need around 3 months for the process of assessment and adoption in the Commission. The projects which are interrupted, on the other hand, have a median duration of 6 months, with the longest interruption leading to up 23 months for the overall duration in one case. When aggregated for all projects submitted for Art 102.2 the median duration is 3.3 months. Furthermore, we also tested whether the process duration differs for projects assisted by JASPERS advisory relative to non-assisted projects, but found no statistically significant difference.

Across sectors, when analysing the median duration of the full process for Art 102.2 by sector, we find a higher duration for projects in Smart development and ICT (5 months) relative to all other sectors (3 months).

In this context, for both types of projects (interrupted and not interrupted), the Commission requests JASPERS IQR to issue a PSA report including an assessment of the quality of the project. We find that, in line with the agreement with the Commission, JASPERS IQR takes around one calendar month (and slightly more during peak periods or holidays) to issue the report. Furthermore, once the report is sent to the Commission, JASPERS IQR also participates in a screening meeting with the Commission services – a meeting where the quality issues for the project are discussed. For the projects interrupted, at the request of the Commission, JASPERS IQR issues also updated PSA reports addressing the replies from the Member State to Commission’s observations. Overall, we assess that JASPERS IQR is highly effective in this process.

Relative to the period 2007-2013, there has been a significant reduction in the overall time taken with the assessment and approval of major projects. This reduction is highest for the procedure based on Art 102.2 (with a reduction of 50%) and lower for the procedure based on Art 102.1 (with a 13% reduction).

By comparison, for the previous period, the REGIO evaluation of JASPERS carried out in 2012 informs us that the average duration of approval for major project was around 9 months, and the minimum duration recorded was 6 months. Therefore, for the current period, we find significant gains in terms of shortening the time of assessment and approval of major projects achieved for the procedure based on Art 102.2 (with a reduction of 50% in the average duration of the process) 137 and, to a lesser extent, for the procedure based on Art 102.1 (with a reduction of 13% in the average duration of the overall process) 138 . In our assessment, for both these procedures, JASPERS IQR plays an instrumental role in ensuring the assessment of the project quality on time in the current period.

The effectiveness of JASPERS IQR to signal issues of project quality in the PSA report is relatively high for most topics, and especially for issues related to project timetable, risk and sensitivity analysis, feasibility study, and cost and benefit analysis.

Next, we also looked at effectiveness of JASPERS IQR in terms of signalling quality issues in the projects for which it issues PSA reports. For this analysis, we collected data from PSA reports for 66 major projects submitted to the Commission and interrupted for quality issues. For these projects we compared the issues raised by JASPERS in the PSA reports with the Commission’s observations in interruption letters sent to the Member States. This data is reported in Table 5.2.13.

Table 5.2.13 Projects with quality issues raised in PSAs and interrupted by the Commission

Note : Data covers 66 major projects interrupted and with PSA reports.

Source : European Commission observation letters and JASPERS’ PSA reports.

Table 5.2.13 reads as follows. First, we identified the number of major projects for which the Commission included observations on a given topic in interruption letters. For example, as regards the project timetable, we identified 12 interrupted projects for which the Commission made observations related to the topic. Next, we looked at for how many of these projects JASPERS IQR had raised critical (column 2) or non-critical (column 3) issues on the same topic. For project timetable, JASPERS IQR raised critical issues in 2 projects, and non-critical issues in all of these projects. The percent ratio calculated in column (6) gives an overall proxy for the effectiveness of JASPERS IQR as it indicates how often the PSA report helps the Commission identify issues of project quality by topic 139 . For project timetable, for instance, we find that, for all projects for which the Commission raised observations on the topic, the PSA reports also reflected issues of project quality in this regard. The same reasoning applies to all topics.

On this basis, we conclude that the effectiveness of JASPERS IQR to signal issues of project quality in the PSA report is relatively high for most topics, and especially for issues related to project timetable, risk and sensitivity analysis, feasibility study, and cost and benefit analysis. For a majority of projects, the topics on which JASPERS IQR raises critical and non critical issues are also reflected in Commission’s observation letters.

Capacity building and horizontal and strategic advice

As mentioned in the previous section, JASPERS’ portfolio of assignments for capacity building and horizontal and strategic advice for 2014-2020 includes 220 assignments, of which 75% are for horizontal and strategic advice, and the remaining 25% are for capacity building.

Starting with capacity building activities, according to JASPERS’ Quality Manual 140 , the objective with these assignments is to provide trainings, workshops, and online forums on topics relevant for the preparation and implementation of projects, as well as to facilitate the dissemination of guides and studies. In terms of approach, the trainings and events organised for capacity building can be either multi-country or tailored to the specific needs of a country.

As explained in the section for relevance, the majority of assignments for capacity building are multi-country assignments, with the most frequent trainings/ workshops organized for topics related to climate change and environment, state aid, and cost-benefit analysis.

So far, while a large majority activities for capacity building were completed and achieved their objectives for delivering trainings for the Member States, such activities have not proven sufficient to trigger a visible ripple effect on the technical capacity of Member States to address effectively issues such as cost benefit analysis, climate change and environment in project preparation.

When we compare the topics of the capacity building activities completed until end 2018 with the quality gaps identified in project preparation during the advisory work with JASPERS and during the assessment of major projects in the Commission, we identify two main issues. First, despite a relatively frequent focus of capacity building activities on issues of cost benefit analysis, climate change and environment, from the Commission’s interruption letters we learn that these issues are still among the most frequent topics for observations to major projects interrupted. Furthermore, from the analysis of the ACNs issued by JASPERS advisory, we also see that JASPERS provides support on cost benefit analysis to 95% of the projects assisted, and on climate issues to 60% of these projects. Therefore, on the this basis, we conclude that the activities for capacity building organised on these topics have not yet proven sufficient to trigger a visible ripple effect on building an autonomous technical capacity for Member States to address sufficiently these issues in project preparation.

Second, both from the documentation of advisory work and from the Commission’s assessments of major projects, we learn that there is still a significant number of projects in need of advisory support also for other topics such as feasibility study and risk and sensitivity analysis, while the activities organised for capacity building on these topics are not that frequent 141 .

Furthermore, at country level, as noted earlier, we find that countries’ reliance on JASPERS’ advisory assistance for project preparation has not been reduced relative to the previous period, therefore suggesting modest capacity building for project preparation in between the two periods. We find that, of 14 Member States supported by JASPERS advisory for project preparation during the previous period, only two Member States (Poland and Czechia) have reduced their relative reliance on JASPERS’ assistance for project preparation. In addition, if quality of project preparation is an indication of technical capacity, then significant remaining gaps in capacity can be inferred also from the rates of project interruptions for most countries 142 . For 9 out of 13 Member States submitting major projects for Art 102.2, the rate of interruption is at least 50%, as summarized in Table A4.6 in Annex 4. Therefore, also the evidence across countries tends to suggest that activities for capacity building do not seem to have had a visible ripple effect on countries’ technical capacity to prepare projects of higher quality without relying on extensive advisory support.

JASPERS has the proven potential to promote more strategic and innovative approaches for technical capacity, but such approaches were not yet mainstreamed at the portfolio level during the period analysed.

Nevertheless, while at portfolio level we could not identify evidence of effectiveness of activities for capacity building, based on in-depth analysis at assignment level we identified several cases in which the approach or the method of delivery of the activity may have a significant potential to contribute more visibly to overall effectiveness, if mainstreamed strategically across Member States, topics of high interest, and time.

As a first example of good practice, we identified the strategic planning for capacity building adopted by Croatia in 2018 which, in our view, has the potential to increase the effectiveness of such activities as they target the most pressing gaps in expertise for project identification and preparation in the country. This experience with JASPERS support in Croatia is presented briefly in Text Box 5.2.1.

Text Box 5.2.1 Strategic approach of capacity building in Croatia

By end December 2018, JASPERS had included 30 assignments for horizontal and strategic advice and capacity building in its portfolio for Croatia. Half of these assignments, however, were subsequently administratively closed. In 2017 the State Audit Office of Republic of Croatia carried out a performance audit on the impact of JASPERS initiative on the projects prepared for application for EU co-financing 143 , and concluded that the added value of horizontal and capacity building support was rather limited due to deficient planning and absence of a clear assessment of needs. Following the recommendations from the audit, beginning with 2018 and in cooperation with the European Commission and JASPERS NCC, the Managing Authority established a capacity building plan based on two main sources : 1) an assessment of needs carried out by the Managing Authority with the intermediate bodies; and 2) gap analyses carried out by JASPERS’ experts for 6 sectors of importance for the implementation of EU funds in the country. This plan is being implemented during the current period.

Further for Croatia, as part of this strategic approach at country level, more recently we also learnt that the country is working with the European Commission and with JASPERS to establish a national IQR service for review and quality assessment of projects prepared for EU funds.

A second example illustrates the “train the trainers” mode of delivery for capacity building trainings. First recommended for JASPERS in the EIB evaluation carried out in 2010, the “train the trainers” approach focuses the training on local trainers who could subsequently disseminate the know how to larger audiences and repeatedly over time within and across countries. This approach was applied by JASPERS for some of the assignments of capacity building in the current period, but it has not been mainstreamed at portfolio level. In our view this approach also has the potential to enhance the effectiveness of capacity building activities as it can help strengthen the local competences and reach much larger audiences. An example of an assignment applying the “train the trainers” approach in Romania is presented in Text Box 5.2.2.

Text Box 5.2.2 “Train the trainers” mode of delivery in Romania

For the programming period 2007-2013, JASPERS provided support for the development of a professional training on environmental impact assessment addressed to national officials. The “train the trainers” approach adopted for this assignment entailed a 2-days training of a nucleus of around 30 trainers who subsequently provided 15 training sessions to additional 400 trainees from regional and local environmental authorities. The training was positively assessed by beneficiaries as it ensured a high outreach and dissemination of knowledge on the topic.

Finally, as a third example we identify as good practice illustrates the economies of scale that could be achieved by coordinated training across countries on a specific EU priority for the implementation of the funds. The example refers to nine training workshops on EU Sustainable Urban Mobility Planning (SUMP) organised by JASPERS in eight countries. The Text Box 5.2.3 provides the details.

Text Box 5.2.3 Coordinated local workshops on EU Sustainable Urban Mobility Planning (SUMP)

During the period September 2016 – February 2017, at the request of Commission services DG REGIO and DG MOVE, JASPERS organized SUMP trainings for local and regional authorities in eight countries: Bulgaria, Cyprus, Greece, Latvia, Lithuania, Poland, Slovenia, and Spain. Initially, JASPERS surveyed the Member States on their interest in capacity building training on this topic and, when realizing that there was a broader interest across countries, it decided to merge all trainings in one assignment in order to ensure consistency and efficiency of delivery. In this context, the training sessions were tailored to countries’ needs by applying, for instance, the “train the trainers” approach in Slovenia, while in Cyprus and Greece the focus was more on awareness raising for decision makers. The one-day workshops were provided by a private consultancy company selected through the EIB’s Advisory Services Framework contract.

For the horizontal and strategic support, based on the analysis of sampled assignments, there are indications that JASPERS’ support helps compensate for lower local technical capacity for guiding and supervising tasks on specialised topics of policy relevance.

Next, for the horizontal and strategic support, JASPERS’ Quality Manual informs us that the main objective for these assignments is to support national strategies on specific issues, river basin flood management plans or integrated urban development. This type of support is usually not project related, and it can be either sector specific or cross-sectoral. As presented earlier in the section for relevance, however, we find that these assignments are highly diverse in their specific objectives (ranging from horizontal tasks to support for national strategies and plans), and therefore less amenable for an assessment of overall effectiveness. Instead, for this type of support, we sampled 10 country specific assignments and analysed them in more detail.

Of the 10 assignments sampled, 7 were completed and 3 administratively closed by December 2018. From the completed assignments, we selected three assignments which can help illustrate the nature and methods of JASPERS’ work in this context. The following presentation of these examples is based on the action completion notes issued by JASPERS for the respective assignments.

First, in an assignment for Bulgaria, JASPERS was requested by the national authority to provide a peer review of several regional Water and Wastewater Master Plans prepared in the context of a larger Master Planning project funded by the World Bank in the country. The national authority had contracted consultancy services for the preparation of the regional master plans, and JASPERS’s task was to review this work and convey its conclusions to the national authority. Upon the revision of the first master plan, JASPERS developed a model that was subsequently replicated by the consultants for all the remaining plans. The assignment was carried out during 2012-2014, and it resulted in several guidance notes issued by JASPERS when reviewing the regional plans. Stakeholders’ interviewed for this assignment rated positively JASPERS’ work, emphasizing the value added of JASPERS’ advice, especially as it relied on its experience in collaborating with the European Commission.

A second example is an assignment from Romania for supporting the planning of EU investments in smart grids. For this assignment, JASPERS was requested to ensure guidance and supervision for the consultancy service contracted to design a framework and methodology for the assessment of investments in smart grids. The outcomes of this work include a complete methodology for project assessment for smart grids, with a definition of eligibility criteria for projects applying for EU funds in this domain, and mechanisms to support the implementation of these investments. The assignment was carried out during December 2013 – February 2014, and JASPERS’ input was appreciated especially as regarded advice concerning the alignment of the methodology with EU requirements and standards.

Finally, a third example coming from Malta illustrates JASPERS’ assistance for the implementation of the national strategy for Research and Innovation (R&I) launched the context of the ex-ante conditionality for smart specialisation for EU cohesion policy. JASPERS’ task was to assist in refining the sub-criteria for the appraisal of projects applying for EU funding for R&I. As part of this assignment, JASPERS provided the Managing Authority with examples of evaluation criteria and sub-criteria for R&I state aid schemes from other Member States with a similar R&I profile, and it helped re-orient the criteria towards project deliverables and outcomes based on the introduction of result indicators. This work was carried out during the period October 2015 – December 2017, and it was appreciated by the beneficiary especially as it also helped ensure alignment with the EU state aid legislation.

In sum, based on the sampled assignments, we conclude that JASPERS succeeded in achieving the objectives at least for 70% of these assignments 144 , and its support helped compensate for lower local technical capacity for guiding and supervision tasks on specialised topics of relevance for EU cohesion policy. Moreover, based on interviews with the stakeholders involved in these assignments, we learnt that JASPERS is valued by beneficiaries especially for its experience acquired across Member States with the planning and assessment of projects for the policy, and for its advice on aligning these investments with EU legislation and standards.

CEF Mandate

Technical advisory for CEF projects

As concerns CEF, as presented in the previous section on relevance, we learnt that 79% (23 out of 29) of the projects supported by JASPERS during its first pilot mandate with CEF were successful in the CEF competitions for the cohesion envelope.

For around 70% of the CEF assignments for works completed until end 2018 JASPERS’ technical assistance was requested primarily around the time of project application. Some of these projects, however, had been assisted by JASPERS for project preparation during the previous programming period (for the ESIF mandate).

For the analysis of effectiveness, as regards the assignments for advising the preparation of these projects, we first looked at their duration and calculated the time it takes to complete an advisory assignment for a CEF project for 21 assignments completed until December 2018 for the first pilot CEF1.1. On this basis, we found that, while the median duration is around 3 months, the assignments vary significantly in this respect. This data is reported by sector in Table 5.2.14, and by type of project in Table 5.2.15.

Table 5.2.14 Duration of JASPERS’ advisory for CEF projects by sector

Sector	Number assignments	Duration assignment (months)
		Median	Max	Min
Rail	18	2	20	1
Roads	3	5	14	4
Total	21	3	20	1

Note: Data covers CEF assignments started in 2015 (earliest) and with completion date in JADE.

Source: JASPERS’ JADE database, December 2018.

Table 5.2.15 Duration of JASPERS’ advisory for CEF projects by type

Type project	Number assignments	Duration assignment (months)
		Median	Max	Min
Works	13	2	15	1
Mixed	6	8	20	2
Studies	2	3	3	3
Total	21	3	20	1

Note: Data covers CEF assignments started in 2015 (earliest) and with completion date in JADE.

Source: JASPERS’ JADE database, December 2018.

From this data we learn that the advisory assignments for CEF tend to last longer for projects in Roads, relative to Rail. From the classification by type of assignments, however, we see that much of the variability is generated by the difference between support for studies and support for works. The median duration is highest for assignments combining support for works and studies, and it is lower for assignments focused either on studies or on works. Nevertheless, compared with studies, we note also that the assignments mixing studies and works have a wide distribution of duration.

Next, in order to understand better JASPERS’ work in the CEF context, we analysed more in-depth 10 sampled assignments for the CEF1.1 pilot, representing one third of JASPERS’ portfolio for this mandate. Table A4.10 in Annex 4 presents the details.

First, in terms of types of projects in the sample, 9 of these projects are for invesments (works in CEF terminology) and one for a feasibility study. As regards the projects for works, two thirds proved successful in the CEF competition. And so did also the project for the study.

Based on the analysis of the documentation for these assignments we found that, except for one project in Slovakia, for all the other projects for works JASPERS’ assistance was requested for review of CEF application and project documentation, and usually around the period of submission and assessment of the application 145 . Moreover, for half of these assignments, when interviewed, stakeholders stated that JASPERS’ support was asked rather late in the application process and therefore had limited scope for added value (this was especially the case for the unsuccessful projects). And given that around 70% of the 19 CEF assignments for works in JASPERS’ portfolio completed until end December 2018 have a duration of at most 5 months over the period for submission and assessment of CEF applications, we infer that for a majority of assignments for CEF1.1 it is likely that there was little to no scope for JASPERS’ advisory to have an impact on project quality in terms of project technical design due to late involvement in the process.

Nevertheless, while the finding above refers to the technical assistance provided by JASPERS in the context of the CEF mandate, we also learnt that 7 of the investment projects assisted by JASPERS at application stage for CEF competitions had also been assisted by JASPERS technical service for the preparation of these projects during the course of the previous programming period in the context of the ESIF mandate.

In 40% of cases, the studies financed by CEF and assisted by JASPERS advisory based on the CEF1 pilot mandate were feasibility studies prepared for investment projects intended for the CEF financial support.

As regards the technical advisory support provided by JASPERS for projects for studies, we found that some of them refer to feasibility studies prepared for CEF investment projects. More specifically, this is the case in 6 out of all the 15 assignments (completed and ongoing) for studies or mixed projects (i.e. works and studies) prepared with the intention to be submitted for CEF financing.

Finally, from the interviews with the stakeholders, we learnt that, given the scope of the assignments, they appreciated JASPERS’ input as regards suggestions for improvement of project application, and especially in terms of alignment with EU legislation and standards.

IPA Mandate

Technical advisory for IPA projects

As mentioned in the previous section, JASPERS’ portfolio for IPA included 102 assignments by end December 2018. Of these, 38% had been completed, 50% were still ongoing, and the rest were administratively closed. Table 5.2.16 presents these details.

Table 5.2.16 JASPERS’ IPA assignments by status

Source: JASPERS’ JADE database, December 2018

For the IPA mandate, 18% of assignments resulted in IPA projects approved for EU-funding.

The advisory work for project preparation constitutes the bulk of this mandate, with 73 assignments completed, closed or on-going for advisory by end December. Nevertheless, when matching the data on JASPERS’ portfolio with data on approved IPA projects from the Commission, we find that only 13 of these portfolio assignments could be matched with 17 approved IPA projects 146 . This implies that, so far, only 18% of all advisory assignments (completed or ongoing) 147 in JASPERS’ portfolio resulted in IPA projects approved.

As regards the submission of the assisted projects for EU funding, although it is informative as regards the contribution of JASPERS’ services to the mobilization of EU funds, it cannot be interpreted as a close proxy for the effectiveness of JASPERS’ technical assistance due several factors as follows. First, it is possible that, at the time when the feasibility study is carried out, the national authorities reach the conclusions that current conditions for the implementation are not optimal and therefore the project is postponed. Second, even if the project is feasible, it is yet to be included in an IPA programme and apply for EU funding. Third, JASPERS explained that there were also situations when the national authorities eventually decided to fund the project from national sources only. And fourth, JASPERS explained also that there could be a time lag between the moment when the project is proposed for EU funding and the time when the decision is made.

Next we calculated the duration of the completed advisory assignments for project preparation. Table 5.2.17 presents the details by sector.

Table 5.2.17 Duration of assignments for project preparation

Sector	Number assignments	Duration assignment (months)
		Median	Max	Min
Energy and solid waste	14	12	55	3
Rail, air and maritime	1	8	8	8
Roads	1	28	28	28
Water and wastewater	12	19	57	15
All assignments	28	18	57	3

Note: Data covers completed advisory assignments.

Source: JASPERS’ JADE data, December 2018

We find that, while the median duration of an advisory assignment for the preparation of IPA projects is 18 months, some of these assignments could continue for as long as 57 months. The sectors with longest median durations are Roads (28 months) and Water and wastewater (19 months).

From the section on relevance, however, we recall that these assignments are not only for investment projects, but they can also assist national authorities in guiding and supervising contracts for technical assistance. Nevertheless, since data in JADE does not inform on the type of assignment from this perspective, we cannot distinguish further what type of assignments required more time in this regard.

Based on an analysis of sampled assignments, there is indication that the effectiveness of JASPERS’ advisory support is limited in cases when the requests for support arrive late in the project preparation cycle.

In order to understand better the work carried out by JASPERS for advisory assignments for project preparation for IPA, we sampled 6 such assignments for further analysis 148 . All the projects assisted were for investments in a new road, and in waste, water and wastewater management systems. Their total costs range from 4 to 45 million euro. Two of these projects were approved for IPA financing, with the remaining 4 were either not submitted (2 projects) or not approved due to cuts in IPA financing.

For these projects we learn primarily that JASPERS advisory can make a difference for project quality when its support is requested early in the process of project preparation. This was the case for 2 of the assisted projects. For other 3 assignments completed 149 , JASPERS’s support was requested at final stages of project preparation and, despite its assessment of significant outstanding issues with project quality, no substantial changes could be made any longer.

Therefore, we interpret this evidence as suggesting that, at least for the sampled assignments, the effectiveness of JASPERS support proved limited for a majority of cases, although this is not attributable to JASPERS itself to the extent that the main reasons were late request for advisory support and reductions in IPA financing for projects. Under these circumstances, however, in terms of customer satisfaction, we also found that the stakeholders generally appreciated working with JASPERS and valued its advice especially as regards the clarification of EU requirements and alignment of applications with EU standards.

IPA horizontal and strategic support

IPA beneficiaries appreciate JASPERS’ horizontal and strategic support especially as it helps raise awareness and familiarize national staff with respect to the management and appraisal of EU funded projects.

As regards the horizontal and strategic support for the IPA mandate, of the 20 assignments in the portfolio, 80% had been completed or were still ongoing in Montenegro, North Macedonia and Serbia by December 2018. Half of these assignments were created for workshops and advisory support for cost benefit analysis and for assistance with national strategies and master plans. The remaining assignments had a variety of objectives such as preparation of project pipeline, analysis and advice at sector level, preparation for the upcoming programming period (2014-2020) etc.

In terms of distribution by sector, 10 of these assignments are for Water and wastewater, and 6 for Rail, air and maritime. As for the duration of assignments completed, it ranges between 8 and 22 months. Further analysis of the effectiveness of these assignments, however, is not feasible due to lack of data.

Nevertheless, while we could not gather sufficient evidence for an overall assessment of the effectiveness of this type of more horizontal support, we include some examples analysed in more detail by way of illustration as follows.

First, we looked at a sampled assignment from Serbia, where JASPERS provided support for the review of terms of reference for the programme on energy efficiency in public buildings. The assignment was part of JASPERS’ mandate of support for the prioritisation and preparation of projects to be financed under IPA 2014-2020 in Serbia. When interviewed, the national authority using JASPERS’ assistance stated that they valued highly this support and especially JASPERS’ advice as regards alignment with EU requirements and standards.

A second example refers to two workshops organised in Montenegro on project management and cost benefit analysis. For these workshops, part of a larger plan for capacity building in the country, we learnt that the beneficiaries found this type of support useful as it helped raise awareness and familiarize national staff with respect to the management and appraisal of EU funded projects.

Stakeholders consultation

In the online consultation, the stakeholders were asked also for the opinion on the extent to which JASPERS had an effect at project level and on horizontal issues. The replies to this question are summarized in Tables A1.10 and A1.11 in Annex 1.

A large majority of stakeholders consider that JASPERS had a significant, and even decisive, effect on project technical specifications and its documentation. They also rated highly JASPERS’ assistance for projects’ alignment with EU and national policies, and for faster project preparation and approval.

At project level, more than 80% of stakeholders considered that JASPERS had a decisive or significant effect on project technical specifications and for improving project documentation. Other aspects appreciated in this regard by the majority of stakeholders include also JASPERS’ effects on project alignment with EU and national policies, and on faster project preparation and approval.

On other project related aspects, however, stakeholders’ opinions on the extent of JASPERS’ effects are more heterogeneous. Around half of the stakeholders perceived significant or decisive effects of JASPERS on helping achieve cost savings at project level, or on improving project implementation. Still for implementation, more than 50% of respondents perceived JASPERS’ effects as limited or none.

As regards effects on horizontal aspects, around 60% of stakeholders perceived decisive or significant effects of JASPERS’ support on increased administrative capacity, and on improved and faster development of national strategies and plans.

5.3EFFICIENCY

For the criterion of efficiency we analysed the planning process for JASPERS’ assignments as reflected by country plans in order to understand the extent to which these activities are planned and implemented in a timely manner. Further, we also explored the work intensity per assignment, calculated relative to the duration of assignments, and its differentiation by type of assignment. This section includes also an analysis of the developments over time of the financial and human resources for JASPERS’ initiative.

The section is organised in two main parts. The first part focuses on the analysis of the planning process and the implementation of assignments over time, while the second part includes the analysis of financial and human resources over time. The main findings are reported in Text Box 5.3.1.

Text Box 5.3.1 Main findings for efficiency of JASPERS’ activities during 2014-2018

PLANNING AND MANAGEMENT OF ASSIGNMENTS

1)Planning: The monitoring system and country plans for JASPERS’ support have improved over time, but some work still remains to be done as regards consistency of data included in the country plans.

2)Planning: A majority of assignments could not meet the initially expected completion dates, and for 80% of them the expected duration had to be extended for significant periods of time.

3)Planning: In a third of the countries supported by JASPERS significant proportions of assignments are administratively closed instead of completed due to a combination of factors. These assignments tend to be protracted for long periods of time before being eventually closed.

4)Planning: The work intensity per assignment varies significantly by type of assignment, with highest median work intensity (60%) for the PSA support for the Commission. Lowest work intensity is observed especially for long assignments which are administratively closed.

5)Planning: Overall, there is scope for a more structured approach of the planning and monitoring of assignments, with well-defined and targeted objectives, clear and realistic timelines, and milestones that enable assessing the progress over time during implementation.

HUMAN AND FINANCIAL RESOURCES

6)The total budget for JASPERS’ ESIF mandate for the period 2014-2018 was 175 million euro, with the Commission’s average contribution of 81% of the total costs. The ESIF mandate represents 97% of the total resources allocated for the initiative.

7)Relative to the previous period, JASPERS has developed significantly its pool of human resources, with the number of professional and support staff increasing by 55% relative to 2013.

8)Preliminary evidence on developments over time indicates a need for further in-depth reflection on possibilities to optimise allocation of work effort and rationalise costs.

For each of these evaluation findings the supporting evidence is as follows.

Planning and management of assignments

For the analysis of the planning and management of assignments we screened the quartely country plans for three sampled countries: Croatia, Poland and Romania. These three countries together represent 49% of all relevant assignments in JASPERS’ portfolio 150 .

Country plans are rolling documents in which the Member States and JASPERS agree to include assignments planned for JASPERS’ support over time as needs arise. They include assignments for technical advisory for projects, and activities for capacity building and strategic and horizontal support. These documents are updated regularly, and JASPERS issues quarterly updated versions of country plans for all countries supported. For the current analysis, we analysed 33 quarterly country plans available for the three sampled countries over the period 2014-2019 151 . The main results presented in the following paragraphs are summarised in Table A4.11 in Annex 4.

The monitoring system and country plans for JASPERS’ support have improved over time, but some work still remains to be done as regards consistency of data in the country plans.

In a first instance, we collected all assignments included in the country plans over time across all available plans over the reference period. In this way we learn, for example, that Croatia has included 153 assignments overall over the period 2014-2019. For Poland, the plans include a total of 138 assignments, while for Romania 231 until 2019.

Of these assignments, we find that a large majority of them are also included in JASPERS’ JADE database, but not all. For Croatia and Poland, we find that around 85% of the assignments included in the country plans are also included in JADE, while for Romania 74% of assignments could be matched between the country plans and JADE. A number of assignments (7% in Croatia and Poland, and 17% in Romania) were present in the country plans, but not included in JADE 152 . When looking at developments over time in the country plans, however, we find that the assignments missing in JADE are for the early years of the programming period. Therefore, on this basis, we conclude that the monitoring of assignments was initially not fully adapted to ensure consistency between the internal database and the country plans, but that the situation was resolved in the more recent years.

Further as regards consistency of information included for assignments in the JADE database and in country plans, for each assignment we compared the data on its status and timeline from JADE with the developments in terms of its presence and status in the quarterly plans over time. On this basis, we find that data in JADE are consistent with the data in country plans for a large majority of assignments. Nevertheless, for a proportion of 14-21% of assignments (depending on the country) we found a number of inconsistencies. For example, there are assignments which appear and disappear in country plans over time without a clear apparent reason while they are maintained in JADE as on-going over the same period. There are also cases of assignments reported repeatedly as completed in country plans, even two quarters after the completion date recorded in JADE 153 . Overall, however, when checking developments over time in country plans, we find that such inconsistencies are no longer present for assignments created starting 2018 in Croatia and Poland, but less so in Romania.

A majority of assignments could not meet the initially expected completion dates, and for 80% of these the expected duration had to be extended by significant periods of time.

Next, still for the assignments that are present both in the country plans and in JADE, we analysed the extent to which these assignments are on track relative to the initial planning established when first created. For this purpose we compared the initial expected completion date with the date of completion/ closure of the assignment for assignments finalised or, if still on-going, with their status by December 2018. Overall, this could be done for a large majority of assignments as country action plans normally include an expected date of completion. Nevertheless, we learnt that the extent to which this information is included in country plans also varies across countries : for Poland, for example, 98% of the assignments in the country plans also have an expected date of completion, while this proportion is lower for Romania (92%) and Croatia (87%).

On this basis, relative to initial planning, less than half of the assignments included in the country plans are on time. More precisely, around 44% of assignments had been completed/ closed on time (or earlier) or were within the planned period if still on-going by end December 2018. For Croatia, however, this proportion is much lower, at 27%, thus indicating that the initial planning was highly imprecise for a large majority of assignments. The median duration of delays relative to the initially expected completion dates ranges in 12-20 months, although there is a high variability across assignments. Across countries, longer delays are observed especially for the on-going assignments.

Further for the analysis of planning over time, we also analysed for how many of the assignments which are already past their initial expected completion date there has been a prolongation of the expected completion rate. We found that this was the case for close to 80% of (the delayed) assignments in Croatia, 86% in Romania, and 90% in Poland. The median length of the prolongation at assignment level is in the range 24-32 months, depending on the country. At this stage, however, we cannot tell whether these assignments are on track to meet the updated deadlines since the monitoring system does not include any indication on intermediate progress in implementation, and the updated deadlines were generally set for 2019-2020.

In reply to our analysis, JASPERS explained that an important factor driving the need to extend assignments over time relates to the delays occurring with the projects on the ground.

In a third of the countries supported by JASPERS significant proportions of assignments are administratively closed instead of completed due to combination factors. These assignments tend to be protracted for long period of time before being closed.

Further in the evaluation, an indication of sub-optimal planning is also represented by the proportion of administratively closed assignments. These are assignments initially planned for JASPERS’ support and which were closed before JASPERS could complete the assignment and issue an action completion note (or a progress development report for CEF). Overall, 9% of the assignments in JASPERS’ portfolio were administratively closed, with more than half of them in Croatia, Poland, and Romania. At country level, however, 8 in 25 countries supported by JASPERS have a proportion of administratively closed assignments higher than 10% in their number of assignments. Table A4.12 in Annex 4 presents this distribution across countries.

In order to understand better the administratively closed assignments, we also analysed their duration by type of assignment. Table 5.3.1 below presents the details.

The median duration for administratively closed assignments is highest for an assignment for project implementation, followed by assignments for horizontal and strategic support, and for advisory for major projects. In fact, across all these assignments, we find that 25% of them had a duration of more than 50 months, thus indicating that they were eventually closed after a long period of time. Moreover, based on the analysis of country plans above, we also found that a large number of the administratively closed assignments had no initial expected date for completion or, if they had one, they were delayed relative to initial planning. On this basis, we conclude that the planning and management of administratively closed assignments were rather approximate.

Table 5.3.1 Duration administratively closed assignments

Type assignment	Assignments	Duration assignment (months)
		Median	Max	Min
Advisory major project	43	31	76	1
Advisory non-major project	10	19	50	2
Review function	3	8	19	4
Capacity building	4	9	14	4
Horizontal and strategic support	38	36	56	2
Implementation	1	43	43	43
All assignments	99	25	76	1

Source : JAPERS’ JADE database, December 2018

As regards distribution by sector, we find that the median duration of administratively closed assignments is longest for Rail, air and maritime (40 months), followed by Water and wastewater treatment (35 months). Across mandates, we find that the assignments administratively closed with longest median duration are for the IPA mandate (52 months), followed by the CEF mandate (37 months).

In terms of reasons for administrative closure, we identified three main reasons as follows: a) an agreement between JASPERS and beneficiary country to combine the respective assignment with other on-going assignments, b) closure of the assignment due to change in context (for example, projects prepared for ESIF but eventually submitted for CEF), and c) a decision of the beneficiary country to remove the assignment from the country plan if deemed no longer feasible.

Based on a more in-depth analysis of 10 sampled assignments which were administratively closed, we find that 3 cases could be assimilated to type a) and b) above in the sense that either the assignment was eventually combined with another one, or that JASPERS had provided significant support for the respective project but the assignment had to be closed due to a change in context (as was the case for a project initially prepared for ESIF and subsequently submitted for CEF). For the remaining 7 assignments, however, we found that, once the assignment was accepted, there had been little follow up on the assignment by the national authorities and eventually the assignment was administratively closed, although after being maintained in the monitoring system for a long period of time (21 to74 months).

The work intensity per assignment varies significantly by type of assignment, with highest median work intensity (60%) for the PSA support for the Commission. Lowest work intensity is observed especially for long assignments which are administratively closed.

Next, we analysed the duration of all completed assignments in JASPERS’ portfolio over the reference period until end 2018. Assignment durations by type of assignment are presented in Table 5.3.2 for all mandates at portfolio level.

Table 5.3.2 Durations of assignments completed until end December 2018

Type assignment	Assignments	Duration assignment (months)
		Median	Max	Min
Advisory major projects	218	18	129	0.2
Advisory non-major projects	77	14	72	1
Review IQR	50	5	15	1.0
Review PSA	125	3	23	1
Capacity building	32	14	57	3
Horizontal and strategic advice	81	21	87	2
Implementation	7	21	67	7
All assignments	590

Note : Data covers completed assignments for all mandates. Data for review IQR and PSA is from SFC2014, and for the remaining assignments from JADE.

Source : JASPERS’ JADE database December 2018, and European Commission’s SFC2014 October 2019.

The median duration of assignments is shortest for the review tasks carried out in the context of procedures based on Art 102.2 (PSA) and Art. 102.1 (IQR) for approval/adoption of major projects. The advisory work for major projects, on the other hand, can take around 18 months (median value) for half of the projects, but was extended even as long as 10 years in one case. In fact, for 25% of the assignments for major projects, the duration is above 35 months. Another type of assignment with an even longer duration is the horizontal and strategic advice. With a median duration of 21 months, this support lasted for more than 38 months in 10% of cases.

In terms of advisory support by sector, we find that the longest median duration is primarily for Water and wastewater (22 months), followed by Roads and Energy and solid waste (18 months). Across mandates, the longest median duration for completed assignments is for ESIF (19 months), followed by IPA (15 months), and CEF (3 months).

From the efficiency perspective, however, it is important to understand that the calendar duration of an assignment does not mean full time work for JASPERS’ experts throughout the period. From the in-depth analysis of sampled assignments, and especially for advisory work, we learnt that it can happen that an assignment is accepted and recorded in the system at a given date and the actual work starts much later when the beneficiary is ready to send the project documentation for review to JASPERS.

An example of a very long duration is an assignment for project preparation in Roads present in the monitoring system for 8 years, and which got seriously delayed due to protracted negotiations with landowners in the country which needed to be finalised before the project application could be prepared and submitted. For this assignment, JASPERS could assist with project preparation at the beginning of the process, but then had to wait a long time for the review of project application until negotiations at national level were finalised.

Furthermore, we also learnt that it is not uncommon that several aspects of support are combined into one long assignment. An example is an assignment which also lasted very long (6 years) which entailed supporting the national authorities for the development of a national master plan for transport that would also ensure compliance with the ex-ante conditionality for cohesion investments in the sector. In this case, JASPERS’ advisory effort was distributed over years, starting with the production of a background paper for strategic planning, and following with several guidance notes at the following stages of the work until the strategy was finalised.

As regards the actual time worked per assignment, JASPERS has started monitoring this aspect in 2017, recording the numbers of hours worked per assignment completed or administratively closed. For our analysis, JASPERS shared with us data for 190 assignments started and completed or closed during the period January 2017 – February 2019 154 . This dataset includes 60% assignments for the review function (IQR and PSA), and the rest for advisory work and horizontal assignments.

Using the hours worked per assignment, we calculated the intensity of work per assignment as follows. First, starting from the calendar duration of the assignment, we assumed that 70% of this duration represents working days and calculated a maximum potential working period per assignment 155 . Second, based on the hours worked reported by JASPERS per assignment, we assumed a standard working day of 8 hours and calculated the number of standard days worked per assignment 156 . Finally, we computed the intensity of effective work as the ratio between the number of standard days worked and the length of the potential working period for the each assignment. This gives us the proportion of time worked in the maximum working time available for the assignment. Table 5.3.3 below presents these results by type of assignment.

Table 5.3.3 Intensity of work per assignment in total working time 157

Note: Data covers 98% of assignments started and completed/ closed over the period Jan 2017-Feb 2019.

Source : JASPERS’ data on hours worked per completed assignment.

For presentational purposes, in Table 5.3.3 we include only the median values for the number of days worked and for the maximum number of working days available for an assignment, while presenting in more details the distribution of the intensity of work per assignment.

From this data we learn, for instance, that across the 45 assignments for advisory for major projects, for a majority of them JASPERS worked up to 51 days per assignment, while the maximum duration of working days for a majority of assignments was 195 days. The median value or the intensity of work for this type of assignment amounts to 30%, although we see that there is a high variability across assignments in this group. At a minimum, there is one assignment for which the actual work represents around 1% of the total working time during the assignment. The is the case of one assignment that lasted for almost 16 months, and the days of work reported are around 9. The assignment was eventually administratively closed. At the higher end, we also have assignments with a very high intensity of work. For advisory of major projects, the assignment with maximum intensity of 152% lasted little more than two months, and the numbers of days worked reported is 75. This is possible, for instance, if several experts work in parallel on the same assignment during a given (usually short) period.

Across types of assignments we learn that the assignments for PSA have the highest intensity of work, a result which can be explained by the fact that the period available to experts to review and assess a major project for the PSA service to the Commission is around 30 calendar days. Therefore, the work effort needs to be concentrated in this relatively shorter period of time. Further, also for the PSA, we see that there can be assignments with low intensity, but these are usually in a minority. More than 80% of assignments for PSA reports have a work intensity of at least 30%.

For the remaining types of assignments a similar interpretation applies, although the reliability of the summary values differs with the number of assignments. For instance, for capacity building, horizontal support and implementation, there are very few completed assignments included in the dataset analysed, and therefore the values are more illustrative. Nevertheless, across assignments, for the advisory work we find that actual work is more concentrated over a calendar period for ESIF assignments for major projects (with median work intensity of 33%), followed by CEF advisory assignments (23%), ESIF advisory for non-major projects (16%), and IPA advisory (10%).

Overall, there is scope for a more structured approach for the planning of assignments, with well-defined and targeted objectives, clear and realistic timelines, and milestones that enable assessing the progress over time during implementation.

Further, when analysing the intensity of work in relation to the duration of the assignment, we find that there is little correlation between them. In Figure 5.3.1 we illustrate the relationship between the two variables for assignments for advisory work, distinguishing between major and non-major projects.

Figure 5.3.1 Duration of assignments and intensity of work – advisory

Source : Based on data for hours worked from JASPERS, Jan 2017-Feb 2019

From the scatterplot in Figure 5.3.1, we see that, while the duration and intensity of assignments for most non-major projects tend to concentrate in the area with relatively lower values for both duration and intensity, for major projects there is much more heterogeneity. As already presented, we can identify assignments with a high number of hours worked during a relatively shorter period of time, but for the majority of the assignments there is not much correlation apparent between duration and work intensity since there are many assignments with varying durations at lower levels of work intensity. Jaspers explained that this is due to the fact that the evolution of assignments is highly dependent on the evolution of the projects on the ground.

A similar graph is presented in Figure 5.3.2, illustrating the distributions of the duration of assignments and the work intensity for the two types of assignments for the review function (IQR and PSA).

Figure 5.3.2 Duration of assignments and intensity of work – review

Source : Based on data for hours worked from JASPERS, Jan 2017-Feb 2019

For the review function, the higher concentration of short assignments is clearly apparent for the PSA function, with the work intensity varying from low to very high. For the IQR version of the review, on the other hand, the distribution is more horizontal across the scale for duration, as assignments range from short to rather long (up to 10 months), while the work intensity also varies. With IQR we note that there is a tendency for lower work intensity to be associated with the longer assignments, although the correlation is influenced by outliers.

We believe that two possible reasons for these distributional differences between the PSA and IQR include the following. First, as noted, the PSA assignments have a clear deadline of 30 days agreed between JASPERS and the Commission. This deadline was introduced since the Commission itself is bound by regulatory deadlines for issuing a decision on a major project submitted, and therefore there was a need for predictability as regarded PSA. As regards IQR, however, there was no regulatory deadline until July 2018, and therefore the process tended to be prolonged for some assignments.

Second, while the PSA service is provided directly to the Commission, for the IQR JASPERS needs to run an interactive process of assessment with the Member State. This happens, for instance, due to the need to clarify certain aspects of the projects with the Member State during the assessment. And these exchanges do take time. When we compared the total duration of the IQR process, including also the notification and approval in the Commission, we found that it takes around 7 months for a majority of projects, and therefore four months more than for non-interrupted projects, and one month more than for interrupted projects in the case of Art. 102.2. In the future, however, we expect that the IQR process will become efficient for more assignments due to the regulatory deadline of 6 months introduced for it in the Omnibus regulation in 2018.

To sum up, based on the analysis in this sub-section, we conclude that there is scope for improvement and rationalisation of the planning and management of assignments included in JASPERS’ portfolio. First, for a third of the countries assisted we find high proportions of assignments included in country plans and subsequently closed administratively, and this type of assignments have very low work intensities and tend to be protracted for long periods of time. Nevertheless, in order to prepare these assignments for approval with the Commission, JAPERS needs to invest resources ex-ante and, given that the value added is minimal to none eventually, this is a source of inefficiency. Second, as regards assignments that are completed and on-going, we find that the planning for a majority of them in the sampled countries was rather imprecise, as they needed to be subsequently extended by significant periods of time. In this regard, we assess that there is a need for more structure in the planning, with well-defined and targeted objectives, clear and realistic timelines, and milestones that enable assessing the progress over time during implementation.

Human and financial resources

The total budget for JASPERS’ ESIF mandate for the period 2014-2018 was 175 million euro, with the Commission’s average contribution of 81% to the total costs. The ESIF mandate represents 97% of the total resources allocated for the initiative.

The financial resources mobilized for JASPERS’ operations for the ESIF mandate amount to 175 million euro over the period 2014-2018, representing 84% of the budget committed for the period. Of these, the Commission contribution covers 81% of the total costs. Additional 1.4 million euro were allocated for the CEF1.1 mandate over the period 2015-2018, of which the Commission contribution covered 90% 158 . Finally, for the IPA mandates, the resources allocated amount to around 4 million euro over the period 2015-2018, representing 64% of the total budget planned for the period 159 .

Relative to the previous period, JASPERS has developed significantly its pool of human resources, with the number of professional and support staff in 2018 increasing by 55% relative to 2013.

In what follows we focus on the ESIF mandate as it represents 97% of the total resources allocated to JASPERS initiative in the current period, and it also enables a comparison with the first period (2007-2013) of JASPERS’ existence. Table 5.3.4 presents the main coordinates in terms of actual human and financial resources allocated to the initiative over the two periods of time.

Table 5.3.4 Human resources and actual expenditure for JASPERS 2007-2018

ESIF mandate	Period 2007-2013	Period 2014-2018
Total staff (average FTEs / year), of which:	79	122
Professional staff (% in total)	85%	85%
Support staff (% in total)	15%	15%
Cumulative total expenditure (million. euro), of which:	186	175
Staff salaries (% in total)	82%	91%
External consultancy (% in total)	15%	6%
Travel and other (% total)	3%	3%
Cumulative EC contribution (million. euro)	132	142
Share EC contribution in total cost (%)	71%	81%
Cumulative actual to planned total cost (%)	n.a.	84%

Note: Professional staff includes JASPERS’ management and the experts.

Source: Based on data reported in JASPERS’ annual reports 2007-2018.

When comparing the resources allocated over the two periods of time, we note that JASPERS has grown significantly over time. If in 2007 it started with 56 employees, in 2018 professional and support staff increased to 140 - therefore 2.5 times higher. As reflected by the developments over time explained in the introductory chapters, this is due to the need to adapt the initiative to the demand for JASPERS’ services, which have expanded and become more diversified in the current period.

Moreover, from the structure of the budget, we also note that there has been a reorientation in JASPERS’ staff policy in the current period, in that the initiative now relies on own staff to a larger extent. In the previous period the use of external consultancy services complementing JASPERS’ expertise was higher, as indicated by the larger share of expenditure with such services contracted by JASPERS. Consequently, we see that the bulk of JASPERS’s resources are currently invested in its people, with the share of staff costs increasing from 82% to 91% between the two periods. As regards conditions of employment, the FAFA agreement establishes that JASPERS’s personnel be employed under the EIB standard employment conditions. In terms of distribution of JASPERS experts across locations in year 2018, 21% were located in Luxembourg, 13% in Brussels, and the remaining 67% across the regional offices in Bucharest, Sofia, Vienna and Warsaw.

As regards the sources of financing, data indicates an increase in the Commission’s contribution to the total budget of the initiative from 71% to 81% between the two periods – an increase which served primarily to cover the higher staff costs incurred during the current period.

Next, based on the two panels in Figure 5.3.3, we look at the dynamics of the human and financial resources, and of the number of assignments over time.

Figure 5.3.3 Human and financial resources, and assignments over time, ESIF mandate

Source: Based on data reported in JASPERS’ annual reports 2007-2018

The left panel in Figure 5.3.3 illustrates the number of annual full time equivalents (FTEs) supported in JASPERS and the annual Commission’s contribution over the period 2011-2018. This data indicates that, starting with year 2014, there has been a systematic increase in staff every year over the period, while the Commission’s annual contribution increased primarily in 2015-2016, and in 2017-2018. Relative to 2013, JASPERS’ personnel was 55% higher and Commission’s annual contribution was 65% higher in 2018.

Preliminary evidence on developments over time indicates a need for further in-depth reflection on possibilities to optimise allocation of work effort and rationalise costs.

To some extent, the right panel in the figure helps us understand better these developments, as it illustrates the dynamics in the number of the on-going assignments by year, together with the evolution of completed and administratively closed assignments. First, as regards the on-going assignments, we note an almost linear increase on a yearly basis until year 2017, while staff continued increasing significantly throughout the current period. The increasing annual number of assignments until 2017 can be explained by the rate of accepting assignments and the median duration of assignments of roughly two years. Assignments’ completion/ closure took off mostly starting with year 2016. The number of on-going assignments, however, is only an imperfect proxy of the actual work on the assignments since, as explained earlier, the work intensity per assignment varies significantly across assignments, and it is not very high for a majority of assignments other than the PSA review function. Nevertheless, a more detailed analysis of this aspect of efficiency is not feasible for this evaluation as it would require data on hours worked per assignment for all assignments and per calendar year– data which was not available for the period analysed in this evaluation. Therefore, we conclude with a partial assessment that this preliminary evidence indicates a need for further in-depth analysis of the topic in order to help identify possibilities to optimise allocation of work effort and rationalise costs.

5.4COHERENCE

In this section we analyse the external and internal coherence of JASPERS’ services. External coherence includes the alignment of JASPERS’ initiative with the EU priorities, with a focus on the objectives of Cohesion Policy and on the higher EU level objectives for climate, environment, and competition in the Union. Further for the external coherence, we also explored the possibility of analysing the synergies between JASPERS and other centrally managed initiatives for technical assistance. As regards internal coherence, we analyse the consistency of approach for project assessment between the two main divisions of JASPERS : advisory and IQR. The main findings are reported in Text box 5.4.1.

Text Box 5.4.1 Main findings for coherence of JASPERS’ activities 2014-2018

Coherence with EU priorities

1)EU priorities: JASPERS’ services are fully coherent with EU requirements and standards for cohesion policy, and with the higher EU level objectives for climate, environment and competition. Its services contribute to guiding project preparation and to supporting the assessment of these projects during adoption in line with these policy priorities.

Coherence with other advisory services at EU level

2)Other EU advisory services: The potential for synergies between JASPERS and other technical assistance initiatives had not been exploited until end 2018.

Internal coherence between the advisory and review functions

3)Internal coherence: JASPERS IQR and JASPERS advisory are broadly consistent in their approach to project assessment, but some scope for further alignment remains. This scope is limited to up to 10% of the projects for the minimum regulatory requirements regarding project quality, and it is wider for non-critical aspects identified with project quality.

For each of these evaluation findings the supporting evidence is as follows.

Coherence with EU priorities

JASPERS’ services are fully coherent with EU requirements and standards for cohesion policy, and the higher EU level objectives for climate, environment and competition. Its services contribute to guiding project preparation and to supporting the assessment of these projects during adoption in line with these policy priorities.

The application for EU financing for a major project for Cohesion Policy requires the Member States to ensure the alignment of the project with EU priorities regarding objectives of Cohesion Policy and the higher EU level objectives of mitigation and adaptation to climate change, account for environmental impacts, and state aid compliance. In consequence, all JASPERS services are organised so that they promote and align the advisory and review work with these EU priorities.

We analysed the coherence of JASPERS’ work with these EU priorities based on the documentation issued by its services, and especially the documents which include the assessments of the projects with respect to objectives of Cohesion Policy, environment, climate and competition law. Table 5.4.1 summarizes the information on the document sections which reflect these assessments in the IQR and PSA reports for the review function, and in the ACNs for the advisory function.

Table 5.4.1 Sections of JASPERS’ documents for assessment of alignment with EU priorities

Source: based on the templates established for IQR and PSA reports, and on JASPERS’ internal guidelines for action completion notes.

For the IQR review, the quality review criteria for the assessment of a major project are established in Commission Delegated Regulation (EU) 480/2014 Annex II. On this basis, as regards the project’s alignment with the objectives of Cohesion Policy, in Section 3 of the IQR report, JASPERS IQR summarises the information on project objective and assesses its consistency with the priority axes of the operational programme from which the resources for investment are to be allocated. For the EU environmental and climate priorities, JASPERS IQR summarizes the information on the analysis of environmental impacts including, for example, the implementation of polluter pays and preventive action principles, compliance with the EIA Directive 160 , ex-ante conditionalities of environmental legislation for Cohesion Policy, compliance with the Habitats Directive 161 etc.

Similarly, for climate adaptation and mitigation measures, JASPERS IQR provides an assessment of project’s contribution to the objectives of the EU climate change policy, and in particular to the Europe 2020 strategy, as well as measures to ensure resilience to climate change variability. For state aid issues, part of the working arrangements for IQR between the Member States and JASPERS IQR is the possibility to ask JASPERS’ experts to consult the Commission on state aid rules applicable to the project. This information is included in the IQR report, where JASPERS explains whether this consultation took place, and whether it considers that the project might involve state aid. Further, in case the project does involve state aid, JASPERS also gives its opinion on whether the state aid is in line with the regulatory requirements. Finally, for all these issues JASPERS IQR needs to provide clear statements whether the project meets the regulatory quality criteria together with supporting justifications.

Further, upon receiving a project notification including also JASPERS’ IQR report, the Commission has the possibility to reject the request for EU funding if it considers that a positive statement in the IQR report is not supported or it is contradicted by information available to the Commission 162 . This could be the case, for instance, if the Commission has additional information on complaints, pilot investigation, infringements, national proceedings, Ombudsman investigation etc. on environmental issues. In our analysis, however, we did not find any such cases where notifications of major projects based on the IQR assessment were rejected by the Commission for non-fulfillment of these quality criteria.

By comparison with the IQR process, the PSA assessment is more focused primarily on the technical and operational aspects of the project. It does not cover the assessment of project’s alignment with the operational programme and neither assesses the compliance with the environmental regulations and EU state aid rules. These aspects are covered by the Commission through its internal inter-service consultation. As regards climate change aspects, the PSA report provides an assessment of the project’s alignment with climate proofing requirements established in the Memorandum of Understanding agreed upon by DG REGIO and DG CLIMA in 2015. This agreement between the two Commission services ensures that JASPERS provides an assessment according to several criteria such as the national adaptation strategy and the project’s contribution to it, the GHG emissions entailed by the project, identification of project vulnerability as regards climate change, relevant measures to ensure resilience etc.

Finally, in the course of advisory work, depending also on the scope of the assignment requested by the Member States, JASPERS advisory provides an assessment (or, at a minimum, a description) of the project’s alignment with the EU requirements on policy objectives and EU priorities for the environment, climate change, and competition rules. In this process, it can raise significant issues which require changes in project documentation in order to ensure alignment with EU requirements and standards or, for less critical issues, it can make comments with suggestions for further improvement. Earlier in this chapter, in the section for relevance, we found that JASPERS’ advice on climate and environmental issues is rather frequent, for 54-60% of the projects assisted, while state aid issues are addressed in about a quarter of these projects.

To sum up, as regards coherence with EU cohesion policy and priorities for the environment, climate, and competition, we conclude that JASPERS’ services are fully coherent with EU requirements and standards, and its service contribute to guiding project preparation and to supporting the assessment of these projects during adoption in line with these policy priorities.

Coherence with other advisory services at EU level

The potential for synergies between JASPERS and other technical assistance initiatives had not been exploited until end 2018.

For this evaluation, we also explored the possibility of assessing the coherence and synergies between JASPERS services and other Commission’s initiatives for technical assistance. Under the current Multiannual Financial Framework (MFF), there are several centrally managed initiatives for technical assistance, and most of them are for capacity building of horizontal nature and for project preparation. They include, for example, the European Investment Advisory Hub (EIAH), the European Local Energy Assistance (ELENA), Connecting Europe Facility (CEF), European Energy Efficiency Fund (EEEF) etc. 163 . Among these initiatives, JASPERS is one of the largest - with 30% of the total cumulated budget of more than 700 million euro for technical assistance centrally managed.

JASPERS’ Quality Manual mentions the possibility for JASPERS to cooperate in sharing expertise with other technical assistance services. Moreover, in 2015, following the publication of the EFSI Regulation 164 , JASPERS analysed the scope for a cooperation with the European Investment Advisory Hub (EIAH) and even established procedures in this regard. In our analysis of JASPERS’ portfolio, however, we did not find any significant work involving such cooperation with other centrally managed initiatives for technical assistance over the reference period of the evaluation. Therefore, on this basis, we conclude that the potential for synergies between JASPERS and other technical assistance initiatives has not yet been exploited during the period analysed 165 .

Internal coherence between the advisory and review functions

JASPERS IQR and JASPERS advisory are broadly consistent in their approach to project assessment, but some scope for further alignment remains. This scope is limited to up to 10% of the projects for the minimum regulatory requirements regarding project quality, and wider for non-critical aspects identified with project quality.

Next, we analysed also the coherence between the advice provided by JASPERS’ advisory to the assisted projects, as reflected in the ACNs issued for this work, and the project assessments carried out by JASPERS IQR in the PSA reports. This analysis covers 77 major projects which were first supported by JASPERS advisory during project preparation, and were subsequently submitted to the Commission for approval based on the procedure of Art. 102.2 with PSA assessment.

As explained earlier, we used the ACNs issued by JASPERS advisory to classify the type of contribution that JASPERS’ support had to the projects by topic. This classification is the same as the one used for the chapter on effectiveness, and it groups projects as follows: 1) projects for which the ACN raises significant issues with project quality, and which require revision before the project is considered suitable for the application for EU financing; 2) projects with no significant issues, but with comments and suggestions for further improvement of project preparation; in such cases the issues are not considered critical from the regulatory perspective of project quality; 3) projects with no significant issues and no comments, but with evidence that JASPERS had contributed to improving project quality, 4) projects for which JASPERS provides a positive assessment on a given topic, but for which we could not find evidence on 1) to 3), and 5) projects for which JASPERS provides a description of the topic, but no explicit assessment.

Further, for the same projects for which we analysed the ACNs, we also screened the PSA reports issued later by JASPERS IQR during the approval process in the Commission. In these reports, JASPERS IQR emphasized critical and non-critical issues still pending with project quality by topic, if applicable. As explained earlier, we found that it is often the case that the assessment made by JASPERS IQR in the PSA reports is also reflected in the Commission’s observation letters sent to the Member States when project are interrupted.

For the analysis of coherence of the two rounds of JASPERS’ assessment during the advisory phase and in the PSA process, we analysed how frequent were the cases where critical issues for a given topic signaled by JASPERS IQR were also raised as significant issues, or comments, by JASPERS advisory. A summary of the data for this analysis is presented in Table 5.4.2.

Table 5.4.2: Comparison of PSA reports and ACNs for projects with critical issues in PSA

Topic with critical issue(s) in PSA report	Number projects	Conclusions in ACN:			% significant in critical
		Significant issue	Assessed, with comment	Assessed, no issues raised
Cost benefit analysis	16	8	7	1	50%
Feasibility study	13	5	6	2	38%
Risk and sensitivity analysis	7	3	0	4	43%
Project description	8	3	1	3	38%
Project costs	9	2	3	4	22%
Capacity of beneficiary	2	2	0	0	100%
Financial plan	2	1	0	0	50%
Project timetable	2	1	1	0	50%

Note: Based on analysis of 77 major projects submitted for Art. 102.2

Source: PSA reports and ACNs

Table 5.4.2 reads as follows. For cost benefit analysis, for example, JASPERS IQR signaled critical issues on this topic in its PSA reports for 16 out of 77 the major projects analysed. When looking at the corresponding ACNs, we find that JASPERS advisory raised significant issues with cost benefit analysis only for 8 of these 16 projects, and included comments for further improvement for 7 of these projects. Therefore, we conclude that JASPERS’ advisory was fully coherent with the PSA assessment in 50% of the cases, and less so for the remaining cases for which it provided comments but no firm warning on outstanding issues. A similar interpretation applies to all topics included in the table.

For a large majority of projects (above 90%), we assess full coherence between the two services in the following sense. First, for projects with no critical issues identified in PSA, there are cases for which JASPERS advisory had raised significant issues during project preparation and the Member State addressed them before submitting the project to the Commission. Therefore, these issues were no longer outstanding when the project reached the PSA assessment, and the two types of assessment are coherent even if different. Second, for other projects for which the PSA did raise critical issues on a given topic, JASPERS advisory also signaled significant issues during project preparation, but they were not addressed by the Member State by the time the project reached the Commission. Also in this case we consider that the two assessments were coherent, as the outstanding critical issues were due to lack of improvements in the application by the Member State despite early warnings.

Nonetheless, this analysis also indicates partial coherence between JASPERS advisory and JASPERS IQR assessment in PSA reports for about 9-10% of major projects for the topics of cost benefit analysis, feasibility study, and projects costs, and to a lesser extent (6%) for risk and sensitivity analysis, and project description.

Furthermore, we carried out a similar analysis also for the non-critical issues identified by JASPERS IQR in the PSA report, and their correspondence with the support from JASEPERS advisory. The data for this analysis is summarized in Table A4.13 in Annex 4. In this case we focused more on the projects for which the ACNs did not signal any further scope for improvement (either in comments or significant issues) while they had non-critical issues identified in the PSA on a given topic, and we found that consistency in assessments differs significantly across topics between the two services. For example, as regards the capacity of beneficiary, the PSA assessment differs from the ACN assessment in 38% of the projects analysed. Similarly, for cost benefit analysis, risk assessment and the project financial plan, the proportion ranges in 30-34%. Therefore, on this basis, we conclude that there is more heterogeneity between the two types of assessments as regards the identification of non-critical issues in project quality.

To sum up, as regards the internal coherence between the assessments produced by the two types of JASPERS’ services, we conclude that the JASPERS IQR and JASPERS advisory are broadly consistent in their approach to project assessment, while some scope for further improvement remains. This scope is limited to up to 10% of the projects for the minimum regulatory requirements regarding project quality, and it is wider for non-critical aspects.

5.5EU ADDED VALUE

This section includes the main findings for the analysis of the EU added value of JASPERS initiative. It relies primarily on the feedback received from stakeholders responding to the online targeted consultation run for this evaluation during summer 2019. In addition, we also present highlights from the perceptions of stakeholders interviewed on aspects of JASPERS’ EU added value for the in-depth analysis of sampled assignments and countries. Finally, the section also includes a summary of JASPERS’ own assessment of the added value of its advisory work. The main findings for the EU added value are presented in Text Box 5.5.1.

Text Box 5.5.1 Main findings for the EU added value of JASPERS’ activities 2014-2018

1)Stakeholder consultation: A large majority of stakeholders consider that, relative to other advisory services, JASPERS’ added value is higher especially as regards its support for the alignment with EU priorities, the potential for knowledge transfer among Member States, its hands-on experience, and its technical expertise.

2)Stakeholder consultation: A large majority of Managing Authorities and project beneficiaries/ promoters consider that JASPERS’ services will still be needed in the future. Willingness to pay from own technical assistance for these services, however, is rather low for both types of stakeholders.

3)Sampled assignments: Further evidence from sampled assignments indicates an appreciation of JASPERS’ added value especially in terms of its technical expertise and EU-wide experience acquired while working in many countries, and with the European Commission.

For each of these evaluation findings the supporting evidence is as follows.

For the analysis of the EU added value of JASPERS initiative, we first asked the stakeholders for their opinions on JASPERS’ added value relative to other advisory services with regards to several service aspects such as the quality of technical expertise, the hands-on experience with projects, the knowledge transfer from other Member States, the knowledge of the context of the beneficiary Member State, etc. 166 . The full list of aspects considered for the question and the distributions of replies by role of respondent (i.e. Managing Authorities, project beneficiary / promoter, consultancy company, or other type of stakeholder) are presented in Table A1.13 in Annex 1.

A large majority of stakeholders consider that, relative to other advisory services, JASPERS’ added value is higher especially as regards its support for the alignment with EU priorities, the potential for knowledge transfer among Member States, its hands-on experience, and its technical expertise.

From the replies to this question we learn that a large majority (more than 80%) of stakeholders appreciate a higher added value for JASPERS services especially with respect to their alignment with EU requirements, the potential of knowledge transfer from other Member States, its hands-on expertise with projects, and the quality of its technical expertise. Slightly more heterogeneous opinions were stated for aspects such as flexibility in responding to needs and expertise on administrative and procedural issues. Also for costs it appears that that up to 60% of stakeholders with an opinion considered that JASPERS services had higher added value relative to other advisory services, although in this case it is less clear how the respondents interpreted the question as, when compared with other advisory services, JASPERS support is free of charge for the beneficiary Member State.

A large majority of Managing Authorities and project beneficiaries/ promoters consider that JASPERS’ services will still be needed in the future. Willingness to pay from own technical assistance for these services, however, is rather low for both types of stakeholders.

Further, stakeholders were also asked to express their opinions on the need for JASPERS services in the future. The question asked and the distribution of the replies from the Managing Authorities and project beneficiaries/ promoters are reported in Table A1.14 in Annex 1.

In this regard, we learnt that at least 70% of the Managing Authorities responding to the consultation consider that JASPERS services will still be needed in the future, even in case when the Commission’s decision is no longer necessary for the approval of major or strategic projects. They also consider that, relative to the review function or horizontal advice, the advisory support will still be needed in the future, at least in some areas. When asked about their willingness to pay for JASPERS services from own resources for technical assistance, however, only 19% agreed they would be willing to do so, while the response rate to the question was also lower.

Similar replies to this question were provided also by the project beneficiaries/ promoters. Compared to the Managing Authorities, even relatively more respondents among project beneficiaries/ promoters agreed that JASPERS support will be needed in the future at least in some areas, and that it will also be needed for assistance with the alignment of national strategies to the EU sectoral policy priorities. The willingness to pay for JASPERS services stated by project beneficiaries/ promoters settled at 28% of respondents, while the response rate was lower also for this type of stakeholders.

Further evidence from sampled assignments indicates an appreciation of JASPERS’ added value especially in terms of its technical expertise and EU-wide experience acquired while working in many countries, and with the European Commission.

In addition, in the context of the in-depth analysis of sampled assignments from JAPERS portfolio and for the sampled countries, stakeholders of the respective projects (Managing Authorities, beneficiaries and staff from the European Commission) were also interviewed about their opinion on the added value of JASPERS services. The results are very similar to the findings from the online consultation, in that stakeholders tend to appreciate JASPERS’ services for their technical expertise, for their experience with similar projects in other Member States, and for their experience in working directly with the European Commission for the alignment with EU priorities and project appraisal. In this context, stakeholders explained that, in many respects, JASPERS’ services are complementary to other advisory services available on the market. Compared with the private consultancy services, for example, they explained that JASPERS’s role is to guide and supervise the work of private experts on project analysis and documentation such as cost benefit analysis and feasibility studies. JASPERS often assists the beneficiaries in defining the terms of reference for contracting private consultancy services. Further, stakeholders also opined that the quality of JASPERS support is very difficult to replace with local expertise due to their acquired extensive methodological expertise and sector experience across Member States – features less easily found on the local market. Finally, from the operational perspective, the stakeholders often emphasized that the use of JASPERS services does not require public procurement procedures, which is considered an advantage.

From the staff of the European Commission interviewed for this evaluation, we learnt that they appreciate the high technical expertise of JASPERS’ experts especially for more complex technical issues in the projects. As regards the PSA service for the review function, they recognised the visible improvement in the timeline of the approval of projects, which is enabled also by the effective project appraisal provided by JASPERS. In addition, the use of the same service for the appraisal of all projects helps ensure consistency in the decision making process.

Finally, for the analysis of the added value, we also considered JASPERS’ own assessment on the added value to the projects assisted. For this purpose, we analysed the value added fiches issued by JASPERS upon completion of an assignment. In the fiches, JASPERS’ experts provide their views on a number of aspects of value added for their assignments 167 . We carried out this analysis for 30 non-major projects completed in the six countries sampled for this evaluation, and learnt that, at least for these projects, the most frequent aspects of value added chosen by JASPERS’ experts refer to improvements in project quality, increased capacity of the beneficiary and assistance with the removal of barriers to realize projects. The least frequent aspect of value added mentioned in the fiches is the facilitation of implementation.

In sum, based on the feedback received from the stakeholders surveyed and interviewed, we conclude that the EU added value of JASPERS services is reflected primarily by their high technical expertise informed also by experience with projects across sectors and Member States, by their experience in working with the European Commission, and by their ability to explain and support the alignment of national projects and strategies to EU policy priorities and standards.

6. Conclusions

In this evaluation we assessed the implementation of JASPERS’ services and its performance over the period 2014-2018 according to five evaluation criteria: relevance, effectiveness, efficiency, coherence, and EU added value. In the following paragraphs we summarize and discuss the main evaluation findings first succinctly at portfolio level, and then more in detail by mandate and type of JASPERS’ service.

JASPERS’ portfolio of services

At portfolio level, relative to the previous period, we found that JASPERS’ activities expanded both in scope and volume during 20014-2018. In terms of scope, while the ESIF mandate has the highest share (close to 90%) of the overall portfolio in terms of numbers of assignments, JASPERS’ services are now supported by two more Commission services, as DG MOVE and DG NEAR joined DG REGIO in financing the initiative for CEF and IPA projects. JASPERS’ services are now available for all Member States and IPA countries. Further for scope, JASPERS has also diversified its portfolio by including a new function for the quality review of ESIF major projects. In volume, during the first five years of the current period JASPERS has already received more requests for service than in the entire period 2007-2013. Among these services, we noted especially an increase in activities for capacity building through multi-country trainings and workshops developed through JASPERS’ Networking Platform.

Technical advisory for ESIF major projects

For technical assistance for the preparation of ESIF major projects, we found that JASPERS’ advisory services remained relevant during the current period, as reflected by more than 400 assignments requested by the Member States until December 2018. Almost one year later, we learnt that 70% of these assignments resulted in major projects adopted by the Commission – projects which mobilized almost 24 billion euro worth of investments from ERDF and Cohesion Fund.

When compared with the previous programming period 2007-2013, we find that JASPERS’ portfolio has a very similar profile in 2014-2020 as regards the high (close to 60%) concentration of assignments in a few countries, and with technical assistance provided to larger projects (compared with the non-assisted projects). As regards the distribution across sectors, the share of the transport sectors in terms of number of assignments is now highest in the portfolio, with the weight of the sector for water and wastewater treatment reduced by 11%.

For the coverage of major projects by JASPERS’ advisory support, relative to the previous period, the share of projects assisted by JASPERS in all major projects submitted to the Commission has increased from 50% to 58%. This is due to the fact that many Member States use JASPERS’ advisory service for a large majority of their major projects. Only two Member States reduced their relative reliance on technical advisory for project preparation – Poland and, to a lesser extent, Czechia.

In terms of the nature of support, when assisting the preparation of major projects, JASPERS’ advisory services provide guidance and supervision most frequently for the preparation of the feasibility study and cost benefit analysis – studies which are regulatory requirements for accessing EU funding for major projects for EU Cohesion Policy. Further, we learnt also that the extent of JASPERS’ support across topics depends on its timing, with more extensive support provided in assignments requested early in the process of project preparation.

The timing of JASPERS’ involvement in project preparation for major projects has a significant effect on its effectiveness. First, the evidence indicates that, when compared to non-assisted projects, the projects assisted by JASPERS advisory have more chances to be adopted without interruptions for quality issues. Second, JASPERS advisory is positively correlated with a shorter duration of project preparation and with faster financial implementation of the projects subsequent to their approval. The effects on project duration and implementation, however, are statistically significant only for projects which had been supported early in the preparatory stage. According to our estimation, these early assignments are likely to cover around 40% of the major projects assisted 168 . For the remaining projects, assistance is likely to be requested at later stages of project preparation, or even at the application stage, with reduced effectiveness.

Even if, ceteris paribus, the projects assisted by JASPERS have a higher chance to be approved by the Commission without interruptions for quality issues, there is a non-negligible proportion (48%) of these projects submitted for Art 102.2. which are still interrupted. We identify several reasons why this can happen. In some cases, the topics for which projects are interrupted had not been covered by JASPERS’ advisory assignments, while in some other cases the Member States did not follow up on the recommendations for improvement indicated by JASPERS’ advisory during the preparation of projects. Nevertheless, there is also evidence that the effectiveness of JASPERS’ advisory in signalling issues with project quality at the preparatory stage also varies across topics. It is highest for the cost benefit analysis and the feasibility study, and lower for topics such as indicators, project description and project timetable. Part of this more limited effectiveness is also the fact that sometimes the experts’ message on certain aspects of the project is not sufficiently strong in order to signal effectively a need for change.

Overall, however, when compared with non-assisted projects, the projects assisted by JASPERS advisory have fewer critical issues of quality raised during the approval process in the Commission, while the period of interruption is rather comparable between the two groups of projects.

Technical advisory for ESIF non-major projects

Also for ESIF non-major project the relevance of advisory support is indicated by the number of requests for such assistance received from the Member States. During the first five years of the current period, JASPERS created at least as many assignments for ESIF non-major projects as for the entire period 2007-2013.

As regards the profile of the portfolio for non-major projects, for the distribution across countries, also for non-major projects there is a high concentration (around 70%) of assignments across 3 main beneficiary countries, and this is roughly similar in magnitude with the experience from the previous period (although the countries concerned are not exactly the same). As regards the distribution across sectors, we noted an increase in the share of assignments for transport and a reduction in the share of the sector for water and wastewater treatment, while the latter still maintains the highest proportion of assignments across sectors.

Further for non-major projects, the evidence gathered points towards an issue with the prioritisation of these assignments. While the initial intention with the JASPERS’ mandate for this type of service was to focus the activities on countries with fewer major projects, or on pilot projects and projects that could be replicated on a much wider scale in the country, there is little evidence that the assignments implemented for non-major projects were aligned with these priorities over the reference period.

For the nature of JASPERS’ assistance, also for non-major projects, JASPERS advisory contributes most frequently with guidance and advice for the feasibility study and cost benefit analysis. In terms of issues still outstanding by the time the advisory assignment is finalised, JASPERS’ comments for improvements are most often related to procurement and implementation, cost benefit analysis, and risk and sensitivity analysis.

When compared to major projects, however, the possibility to evaluate the effectiveness of the advisory support provided to ESIF non-major projects was much more limited due to difficulties encountered with the data collection for these projects. For example, the non-major projects assisted by JASPERS had a rate of success in the range 50-58% during the approval process with the Managing Authorities in two countries, but we cannot generalise this result since the data collection for other Member States did not prove feasible. In this regard, over the reference period for this evaluation, JASPERS had very little follow up information on non-major projects assisted as it did not collect any data on what happened to them once the advisory work was completed. In our view, this would be necessary in the future in order to have the possibility to assess the effectiveness of such assignments and identify the scope for a better calibration of this service.

Technical advisory for CEF projects

JASPERS’ technical assistance for CEF projects was initiated as a pilot in the current programming period. It started with a first mandate (CEF1.1) valid until March 2018, and it continued with a second mandate (CEF1.2) until the end of 2021. This evaluation covered primarily the first mandate, which applied to projects prepared with the intention to submit them for CEF competitions for cohesion countries.

Within the scope of JASPERS’ first CEF mandate until 2018, JASPERS supported 29 CEF projects in 6 cohesion countries, with the number of project supported being determined by the budget allocated for this pilot.

As regards the types of CEF projects supported by JASPERS advisory, when compared with the non-assisted CEF projects, the assisted projects were more concentrated on works (rather than studies), and tended to be much larger in terms of total eligible costs.

Still for the CEF projects included in the first mandate, we concluded that, while 80% of the supported projects proved successful in CEF competitions, the probability that the advisory had a significant effect on project quality beyond project documentation through these assignments is not high. This is due to the fact that, for a majority of them, the advisory support was requested primarily around the time of CEF competitions and the subsequent assessment of applications. JASPERS informed us, however, that in some of these cases the projects had been assisted by their advisory service for project preparation during the previous programming period, at a time when the beneficiaries were considering submitting the projects for financing from the EU Cohesion Policy. Therefore, we can infer that, at least for these cases, the incentive to request further extensive support during the current period was reduced.

Finally, during the final stages of this evaluation we also learnt that, following the CEF pilot experience with JASPERS’ assistance, DG MOVE has decided to support a renewed mandate for JASPERS’ services for CEF projects during the programming period 2021-2027 – a mandate with a significantly higher budget and therefore higher potential for more extensive assistance to projects in all Member States. In this context, JASPERS assistance for CEF projects will be expected to prioritise early phases of planning and project preparation in order to ensure maximum added-value.

Technical advisory for IPA projects

The IPA mandate was piloted during the last years of the previous period and subsequently developed into a fully-fledged line of advisory service in the current period. During the reference period 2014-2018, JASPERS received requests for advisory support for around 70 assignments. Of these assignments, 17 resulted in projects approved for IPA funding, representing 7% of all IPA projects approved until summer 2019. Therefore, also for this mandate, we concluded that the scope for advisory services was lower than in the case of ESIF major projects. This is also reflected by the much lower budget allocated for this type of service within the overall budget of JASPERS.

In terms of geographical and sectoral distribution of the advisory work for IPA projects, more than half of the assignments are concentrated in two countries, and they are most frequently requested for the sector of water and wastewater (45% of all IPA advisory assignments).

For the nature of support provided, in contrast to the mandates for ESIF and CEF where the focus is primarily on investment projects, the IPA projects supported by JASPERS and approved are mostly for technical assistance rather than works. This is in line with the overall distribution of IPA projects, whereby a large majority of them are financed for technical assistance intended to support the administrative capacity in these countries.

As regards effectiveness, based on an analysis of sampled assignments, also for this mandate we found indication that the effects of JASPERS’ advisory support in terms of project quality and beyond improvements in project documentation are likely to be limited, and for similar reasons. This analysis indicates that the requests for support came late in the process of project preparation, and mostly at application stage.

In sum, as regards the relevance and effectiveness of JASPERS’ advisory function, we conclude that, while this line of service remains relevant especially for the ESIF mandate, there is also scope for further reflection on a more strategic prioritisation of this service. For all mandates, JASPERS’ contribution is likely to be effective when its support is started at early stages of project preparation – a practice which is not yet dominant in JASPERS’ portfolio. Moreover, for the ESIF non-major projects, the support should be more focused on the priorities established initially for this type of service. We also identified a need for better monitoring of the effectiveness of this type of service over time.

ESIF review function

Introduced to respond to the new regulatory requirements for the approval of ESIF major projects for EU financing, the new review function for the ESIF mandate has proven a valuable support in the Commission’s decision making process for the approval of major projects. Overall, relative to 2007-2013, there has been a significant reduction (50%) in the time needed for the approval of major projects – a process in which JASPERS provides timely and effective support for project assessments.

Of the two main strands of the review function (IQR and PSA), we could evaluate the PSA process extensively due to the possibility to collect structured data on process and performance. On this basis, we concluded that JASPERS IQR is highly effective in identifying critical issues in projects submitted for EU financing (when applicable), and in providing its assessment to the Commission in a timely manner. This is reflected by the close correspondence between the observations raised by the Commission in its interruption letters to the Member States and JASPERS’ PSA reports, by the active role of JASPERS IQR in the screening meetings with the Commission services, and by their systematically prompt response to Commission’s requests for post-submission assessments.

As regards the IQR strand of the review function - it also contributes to an overall reduction in the time needed for the assessment and approval of major projects, but we could not evaluate further its effectiveness in terms of critical issues identified for the projects assessed. This is due to the fact that, over the reference period, the IQR process was organised in a much less formal way and there was little scope for collection of structured data in this regard. In terms of more recent developments, in its comments for this evaluation, JASPERS IQR informed us that, starting with late 2018, also the IQR process has been formalised by introducing a structured template for its first assessments of projects submitted for the IQR quality review.

Capacity building and horizontal and strategic support

While this line of service was present in JASPERS’ portfolio also during the programming period 2007-2013, for the current period it has expanded, especially as regards activities for trainings and workshops on topics highly relevant for EU Cohesion Policy. Between the two periods, the share of such assignments has grown from 4% to 25% in JASPERS’ portfolio of activities for capacity building, and primarily for the ESIF mandate.

The assignments for training and workshop activities are organised most frequently (close to 60%) in a multi-country format, often in cooperation with Commission staff, and for topics such as climate change adaptation and mitigation, environmental impacts, state aid, and cost benefit analysis. For country specific trainings, on the other hand, we concluded that the demand was more limited over the reference period, as reflected by the relatively lower number of such assignments requested by the Member States and their distribution across countries.

As regards the assignments for horizontal and strategic support, they were provided primarily for the ESIF and the IPA mandates, and they were more geared towards country specific support of horizontal nature such as sectoral analysis, preparation of project pipeline, preparation of operational programmes etc. For the assistance with more strategic issues, approximately 30%, this type of support was requested by Member States for preparing national and regional strategies for investments for EU Cohesion Policy.

Geographically, based on the distribution of country specific activities for capacity building (including both training activities and horizontal assignments), there is a high concentration of activities (60%) in three Member States for the ESIF mandate – a situation similar with the advisory function.

In terms of timing, two thirds of these assignments had started at the beginning of (or before) the programming period 2014-2020, as they were meant to support the preparation for the upcoming period.

As regards effectiveness, we concluded that, for a large majority of activities for capacity building and horizontal and strategic support, their specific objectives at assignment level had been achieved. Nevertheless, we could not find any convincing evidence that, over a medium to longer term, these activities also had a noticeable ripple effect in terms of strengthened technical capacity in beneficiary countries for identification, preparation and submission of projects for EU funding. On the contrary, in our assessment, the continuing high reliance on JASPERS’ advisory assistance across many of the beneficiary countries, as well as the frequency of quality issues still occurring in the process of approval of major projects, indicate a rather limited impact of capacity building activities on enhanced relevant technical capabilities in the beneficiary countries.

In this respect, however, we also noticed what appeared to be a paradox for the evaluation. On the one hand, we found that the evidence collected did not support the hypothesis of significant impacts of JASPERS’ activities on technical capacity in the beneficiary countries. On the other hand, however, we learnt from stakeholder interviews and from the targeted online consultation that there is a widespread perception among stakeholders that JASPERS’ support does help strengthen technical capacity. Stakeholders also tend to emphasize the value of JASPERS’ hands-on support for technical capacity during the assignments (relative to training).

In our view, this apparent paradox can be explained by the fact that, while JASPERS’ support does help alleviate the lack of sufficient technical capacity in the beneficiary countries, this approach does not prove sufficient in ensuring a long lasting effect at local and national levels. This can be due to several factors. First, as regards the deployment of JASPERS’ services, there is no substantive evidence of an overall strategic approach for building relevant technical capacity across Member States over the reference period. Second, we learnt that, while carrying out the assignments, JASPERS often guides and supervises the work of service providers for technical assistance – an approach which has less potential for a long term impact on capabilities in national authorities in charge with the management of EU funds. Finally, we also found that staff turnover in national administrations is sometimes also a factor not conducive to long term building of technical capacity at national level. In reply to our analysis, Jaspers explained that, for the future, the scope and the strategic approach for activities for capacity building will be strengthened.

Nevertheless, as explained next for the overall stakeholders’ perceptions of JASPERS’ services, we learnt that an undisputed and appreciated contribution of JASPERS with this type of service is their ability to explain and inspire the alignment of projects with the EU requirements and standards.

Finally, as regards support for capacity building, we found also that JASPERS has proven potential for a more strategic and innovative approach to capacity building. Good practice includes examples of strategic planning of such activities at country level, of training activities organised in the “train the trainers” format with a high potential for knowledge dissemination, and of effective organisation methods applied when organising training activities on topics of relevance for several countries. Therefore, we concluded that, while not yet mainstreamed at the portfolio level, such practices have a high potential for a more impactful support for technical capacity. In terms of more recent developments, in its comments to this evaluation, JASPERS informed us that efforts to mainstream such practices at portfolio level are currently under way.

Stakeholders’ views on the relevance and effectiveness of JASPERS’ services

For this evaluation, stakeholders’ feedback was gathered systematically through the online targeted consultation – a survey for which we gathered 210 replies from Managing Authorities, projects promoters and beneficiaries, professionals from consultancy companies, and colleagues from the Commission 169 . Additional opinions on JASPERS’ activities were gathered through interviews for the background analysis of sampled assignments and countries.

On this basis, we learnt that JASPERS’ services are highly appreciated by a majority of stakeholders with varying experiences across mandates and types of service. Almost all of these stakeholders consider JASPERS as relevant or highly relevant for providing technical support for increasing project quality, for assisting with project alignment with EU requirements and standards, and for enabling the development of skills for cost benefit analysis in line with the EU guidelines. Moreover, the stakeholders surveyed consider that JASPERS has a high ability to respond to needs related to these aspects of project preparation. More heterogeneous replies in this regard were noticed for issues related to compliance with EU public procurement and assistance for project implementation.

Further, as regards the effects of JASPERS’ services on a variety of aspects, more than 80% of stakeholders opined that JASPERS had a significant, and even decisive, effect on the technical specifications and documentation of projects. They also rated highly JASPERS’ effects on alignment of investment projects with EU and national policies, and on faster project preparation and approval. More heterogeneous opinions among stakeholders were expressed as regards JASPERS’ effects on achieving cost savings at project level, and as regards project implementation.

Finally, as regards customer satisfaction, a very large majority of stakeholders rated highly JASPERS’ services in terms of their consistency with EU requirements and standards, quality of advice, effectiveness and timeliness.

Efficiency of planning and work allocation

For the criterion of efficiency, the evaluation focused primarily on the planning process of JASPERS’ work, on the rate of completion of assignments and administrative closures, on the allocation of actual work effort during the calendar duration of assignments, and on the dynamics over time of human and financial resources allocated to the initiative.

In terms of planning, we found that, similar to the situation in the previous programming period, JASPERS’ portfolio of activities develops in line with the requests from the Member States. Assignments are included in the planning as they arrive, without a clear process of prioritisation. As explained by type of service above, there is little evidence for a strategic approach in focusing the deployment of JASPERS’ services for activities with highest potential leverage in terms of added value for the management and implementation of EU funds at country level. For example, this is reflected by the fact that concentration of assignments in a few beneficiary countries emerges systematically across periods, mandates and types of services. In our view, for such countries which tend to have a high demand for all types of JASPRS’ services, a more strategic and selective approach in aligning capacity building activities and advisory support could have a higher potential for a more effective technical assistance at country level, possibly reducing also their reliance on assistance on longer term.

This approach in planning also has consequences for the evolution of JASPERS’ work over time, as reflected by several developments. First, we found that the proportion of assignments which had to be administratively closed without fully achieving their objectives (or even not at all) is relatively high in a number of Member States. This happens most likely because the initial planning for such assignments was not sufficiently defined, and they were included in the country action plans without clear expected durations and milestones for monitoring the progress in the interim. As a result, they were protracted for long periods of time until a decision was eventually taken in the Member State to close them.

Second, for the assignments which achieved their objectives, we found that a majority of them were delayed relative to their initial planning, and had to be extended for significant periods of time. In their comments to this evaluation, JASPERS explained that the assignments for advisory usually follow the timeline of projects established in the Member States. Therefore, the delays experienced are primarily due to unexpected developments in the Member States which delay the projects themselves. Nonetheless, while we see the merit of this argument, we also consider, however, that a more structured approach to planning, with an effective mechanism that ensures timely implementation of assignments would benefit the overall efficiency of the initiative. In this regard, in terms of more recent developments, JASPERS informed us that it is currently working on an improved system for planning the assignments and for monitoring the interim progress with the their implementation.

And third, the impact of the tendency to accumulate assignments in JASPERS’ portfolio without prioritisation on efficiency can also be inferred from the analysis of the work intensity per assignment. Based on a sample of assignments implemented over the period 2017-2019, we learnt that the work intensity per assignment relative to its calendar duration varies significantly across assignments. We found that assignments for the PSA quality reviews are most efficient, as they are clearly targeted in terms of objectives and duration. For the remaining assignments, however, there is significant variability in the work intensity per calendar duration primarily due to the time lags in the interaction between JASPERS and their counterparts in the Member States, as explained above.

As regards the assessment of the extent of the scope for improvement in the allocation of work effort across assignments, however, we could not produce an estimate in this evaluation as this would have required data for the hours worked per year per assignment, for all assignments in JAPERS’ portfolio – data which was not available for the evaluation. We learnt, however, that, following the ECA performance audit published in 2018, JASPERS has engaged in a process of improving the monitoring system for its activities, including also tracking the work effort per assignment. Therefore, in our assessment, as more relevant monitoring data accumulates over time, an analysis of the distribution of the work effort by mandate, type of service, sector and country would contribute significantly to identifying the scope and avenues for the optimisation of the planning and allocation of resources within the organisation.

In terms of financial resources allocated to the initiative, relative to the previous period, the overall contribution of the Commission has increased from 71% to 81% of the total expenditure with the initiative, reaching close to 150 million euro for the three mandates over the period 2014-2018. This increase in financial resources has a direct correspondence in the consolidation of the pool of professional expertise employed by JASPERS, with a higher emphasis on own expertise rather than contracted services. As regards the evolution over time, the resources allocated to the initiative evolved in line with the accumulation of assignments in JASPERS’ portfolio.

Finally, as our awareness and knowledge of evolutions over time depend to a large extent on whether we measure them, for this evaluation we conclude also that the experience with JASPERS’ development over time is valuable also as regards the realisation of the usefulness of a better-tuned monitoring system. During the course of this evaluation, we learnt that JASPERS’ processes and performance were gradually improved over time in line with efforts to improve the monitoring system. Although we found evidence that there is still some scope for further developments and fine-tuning, we learnt that much has already been achieved especially starting with 2018. Data quality for the latest years has improved, thus enabling a more informed and better implementation of processes, and there are also ongoing efforts for calibrating better the result orientation of JASPERS’ services.

External and internal coherence

With respect to external coherence, the evidence collected for the evaluation indicates that JASPERS’ activities are fully coherent with EU requirements and standards for EU Cohesion Policy, and with the EU level objectives for climate, environment and EU competition policy. Its services contribute by guiding project preparation, by supporting the project assessments, and by disseminating knowledge in line with the EU policy priorities.

As regards internal coherence of JASPERS’s services, we analysed the consistency of opinions issued by JASPERS IQR and JASPERS advisory on the same ESIF major projects, and found that these opinions were broadly consistent between the two services. Nevertheless, the evidence indicates also that there is still some scope for further alignment for around 10% of the projects in terms of minimum regulatory requirements for project quality, and for a larger number of projects for issues considered non-critical from the regulatory perspective.

EU added value

The EU added value of JASPERS initiative was evaluated based on feedback gathered from the stakeholders surveyed with the targeted online consultation. On this basis, we learnt that, relative to other advisory services, JASPERS’s added value is most evident with respect to the alignment with EU priorities, the potential for knowledge transfer across Member States and pre-accession countries, and its hands-on technical experience. JASPERS’ experts are appreciated especially for their experience accumulated while working on a variety of projects across sector and Member States, and with the European Commission. In this context, stakeholders interviewed further explained that, in many respects, JASPERS’ services are complementary to other advisory services available on the market, considering also that JASPERS’ EU wide experience and methodological expertise would be rather difficult to replace at local level.

As regards the need for JASPERS in the future, a large majority of the stakeholders surveyed considered that the advisory assistance would still be needed in the future, at least in some areas. Nevertheless, when asked about their willingness to pay for these services from own technical assistance, only 19% of the Managing Authorities, and 28% of the projects beneficiaries / promoters, stated that they would be willing to do so.

In conclusion, we consider that this evaluation of the implementation and performance of JASPERS initiative indicates that the initiative contributes to a better planning and implementation of EU funds, it helps promote EU requirements and standards across countries, it has a significant potential for EU added value due to its cumulated knowledge and experience across the Union, and it is highly appreciated by its beneficiaries. Further, we consider that the initiative can work better by calibrating further its effectiveness, efficiency, and internal coherence, while acknowledging also that clear efforts in this regard are currently under way.

REFERENCES

Legislative texts

Regulation (EU) No 1303/2013 of the European Parliament and of the Council of 17 December 2013 laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, the European Agricultural for Rural Development and the European Maritime and Fisheries Fund and laying down general provisions on the the European Regional Development Fund, the European Social Fund, the Cohesion Fund and the European Maritime and Fisheries Fund and repealing Council Regulation (EC) No 1083/2006 (Common Provisions Regulation)

Regulation (EU) 2015/2017 of the European Parliament and of the Council of 25 June 2015 on the European Fund for Strategic Investments, the European Investment Advisory Hub and the European Project Portal and amending Regulation (EU) No 1291/2013 and (EU) No. 1316/2013 (EFSI Regulation)

Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on the financial rules applicable to the general budget of the Union, amending Regulations (EU) No 1296/2013, (EU) No 1301/2013, (EU) No 1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 1316/2013, (EU) No 223/2014, (EU) No 283/2014, and Decision No 541/2014/EU and repealing Regulation (EU, Euratom9 No. 966/2012 (Omnibus Regulation)

Commission Delegated Regulation (EU) No 280/2014 of 3 March 2014 supplementing Regulation (EU) No.1303/2013 of the European Parliament and of the Council laying down common provisions on the European Regional Development Fund, the European Social Fund, the Cohesion Fund, the European Agricultural for Rural Development and the European Maritime and Fisheries Fund and laying down general provisions on the the European Regional Development Fund, the European Social Fund, the Cohesion Fund and the European Maritime and Fisheries Fund

Commission Implementing Regulation (EU) No 1011/2014 of 22 September 2014 laying down detailed rules for implementing Regulation (EU) No1303/2013 of the European Parliament and of the Council as regards the models for submission of certain information to the Commission and the detailed rules concerning the exchanges of information between beneficiaries and managing authorities, certifying authorities, audit authorities and intermediate bodies

Commission Implementing Regulation (EU) 2015/207 of 20 January 2015 laying down detailed rules implementing Regulation (EU) No 1303/2013 of the European Parliament and the Council as regards the models for the progress report, submission of the information on a major project, the joing action plan, the implementation reports for the Investments for growth and jobs goal, the management declaration, the audit strategy, the audit opinion and the annual control report and methodology for carrying out the cost-benefit analysis and pursuant to Regulation (EU) No 1299/2013 of the European Parliament and of the Council as regards the model for the implementation reports for the European territorial cooperation goal

Directive 2011/92/EU of the European Parliament and of the Council of 13 December 2011 on the assessment of the effects of certain public and private projects on the environment (Text with EEA relevance)

Council Directive 92/43/EEC of 21 May 1992 on the conservation of natural habitats and of wild fauna and flora

Other references cited in text

European Investment Bank (2010) – Evaluation of the EIB Role in the JASPERS initiative, December 2010.

( https://www.eib.org/en/publications/evaluation-of-the-eib-role-in-the-jaspers-initiative )

DG Regional and Urban Policy (2012) – JASPERS Evaluation. Final Report 13 December 2012, evaluation carried out by AECOM and Economics for the European Commission. ( https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2012/jaspers-evaluation )

Long, Scott J. and Freese, Jeremy (2013) – Regression Models for Categorical Dependent Variables Using STATA, Stata Press Publication

DG Regional and Urban Policy (2014) – Guide to Cost-Benefit Analysis of Investment Projects. Economic appraisal tool for Cohesion Policy 2014-2020, European Commission, December 2014

( https://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf )

Republic of Croatia State Audit Office (2017) – Impact of the JASPERS programme on submitted project proposals by the Republic of Croatia for co-financing from EU funds. Report on performance audit, Zagreb, May 2017

( https://op.europa.eu/webpub/eca/synthesis-report-jaspers/en/ )

European Court of Auditors (2018) – Joint Assistance to Support Projects in the European Regions (JASPERS) – time for better targeting, Special Report 01, 2018 ( https://www.eca.europa.eu/Lists/ECADocuments/SR18_01/SR_JASPERS_EN.pdf )

JASPERS (2018) - Quality Manual, version 1.2, 2018

European Court of Auditors (2020) – The European Investment Advisory Hub – Launched to boost investment in the EU, the Hub’s impact remains limited, Special Report 12, 2020.

( https://www.eca.europa.eu/en/Pages/DocItem.aspx?did=53644 )

Annex 1: Replies to the online targeted consultation

Figure A1.1

Note : 210 respondents

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.1 Respondents to stakeholders’ consultation by country

Country/ institution	Number respondents	% in total
European Commission	9	4%
Austria	2	1%
Bulgaria	19	9%
Croatia	14	7%
Czechia	11	5%
France	1	0.5%
Germany	1	0.5%
Greece	3	1%
Hungary	6	3%
Italy	1	0.5%
Latvia	5	2%
Lithuania	5	2%
Malta	5	2%
Montenegro	5	2%
Poland	45	21%
North Macedonia	9	4%
Romania	35	17%
Serbia	4	2%
Slovakia	17	8%
Slovenia	2	1%
Spain	5	2%
Turkey	4	2%
United Kingdom	1	0.5%
Multi country	1	0.5%
Total	210	100%

Note : 210 respondents

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Figure A1.2

Note : 210 respondents

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.2 Respondents’ experience with JASPERS services, by role and service type

Role of respondent	Major project	Non-major project	Horizontal and strategic	Capacity building
Managing Authority	56	29	24	19
Project promoter/ beneficiary	80	33	23	15
Consultancy providing TA	19	6	5	3
Other	27	16	9	13
Total	182	84	61	50
of which, declaring ESIF experience	149	65	52	42

Note : 210 respondents overall ; multiple answers possible

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.3 Respondents’ experience with JASPERS services, by role of respondent and service

Note : 210 respondents overall ; multiple answers possible

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.4 Respondents’ experience with JASPERS services by role and topic of support

Note : 210 respondents overall ; multiple answers possible

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.5 : Respondents’ experience with frequency of interaction with JASPERS, by role

Note : 210 respondents overall; multiple answers possible

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

RELEVANCE

Table A1.6 Reasons for asking for JASPERS TA’s support by type of respondent

Note: 210 respondents overall, multiple answers possible

Question: “Which were the motivations for asking / suggesting JASPERS’ support?”

Source: Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.7 Respondents stating that JASPERS is relevant/ very relevant for needs by topic

Note : 210 respondents

Question : “Which of the following needs that may be addressed by JASPERS are relevant in your case ? “ Options for reply : Very relevant, Relevant, Not very relevant, Not relevant, No opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019.

Table A1.8 Respondents’ opinions on JASPERS ability to respond to needs by topic

Note : 210 respondents

Question : “Based on your experience, to what extent do you think JASPERS was able to respond to the following needs.” Reply options: Very large extent, Large extent, Limited extent, Very limited extent, No opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.9 Respondents’ satisfaction with JASPERS’ services by topic

Note : 210 respondents

Question : “Based on your experience, to what extent are you satisfied with the following aspects?” Reply options: Very large extent, Large extent, Limited extent, Very limited extent, No opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

EFFECTIVENESS

Table A1.10 Project related effects of JASPERS, by role of respondent and topic

Note : 210 respondents

Question : “Based on your experience, what do you think was the effect of JASPERS on the following aspects?” Reply options: Decisive, Significant, Limited, Neutral, Negative, No opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.11 Strategic and horizontal effects of JASPERS, by role of respondent and topic

Note : 210 respondents

Question : “Based on your experience, what do you think was the effect of JASPERS on the following aspects?” Reply options: Decisive, Significant, Limited, Neutral, Negative, No opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

EU ADDED VALUE

Table A1.12 Familiarity with and use of other advisory services

Other advisory services	Used the service	Familiar but not used the service	Total replies
Structural Reform Support Service	11	10	207
EIB advisory services (other than JASPERS)	64	41	207
The World Bank advisory services	51	40	207
EBRD advisory services	34	41	207
Private consultacy services	151	25	207

Note: Overall, 172 respondents used at least one of these services. Only 21 of the respondents are familiar with at least one of these services but used none.

Question: “Please indicate your familiarity with alternative support schemes (EIB advisory, World Bank advisory, EBRD advisory, Private consultancy, Structural Reform Support Service).” Reply options: I am familiar with the following alternative scheme (Yes/No), I have benefitted from support from the following alternative scheme (Yes/No).

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.13 Added value of JASPERS’ services in comparison with other advisory services

Note : 207 respondents

Question: “For the mandates and activities you are familiar with, what do you think is the added value of the following JASPERS services as compared to possible alternatives (e.g. private consultancy, in-house advisory, other)? Reply options: higher, same, lower, no opinion.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Table A1.14 Opinions on the need for JASPERS services in the future and willingness to pay

Note : Respondents include 62 Managing Authorities and 96 beneficiaries/ project promoters. Numbers reported in cells: proportion of replies “agree to a large of very large extent”, and number of all opinions in parantheses.

Question: “To what extent do you agree with the following statements?” Reply options: very large extent, large extent, limited extent, very limited extent.

Source : Stakeholders’ consultation for JASPERS evaluation, June – September 2019

Annex 2: Data and regression models

A2.1 Data collected for the evaluation

Table A2.1.1 : Summary of data for relevance

Table A2.1.2 : Summary of data for effectiveness

Table A2.1.3 : Summary of data for efficiency, coherence, and EU added value

A2.2 Models

M1. Model for the use of CPR Art 102.2 for submission of major projects

(Section 5.1 Relevance, ESIF mandate, Review function)

Table A2.2.1: Variables used in the model:

Variable	Values	Share of "yes"
Projects submitted for 102.2 (dependent variable)	1=yes, 0=no	71%
Jaspers' TA assistance	1=yes, 0=no	67%
Year submission major project
Year submission 2015-2016	1=yes, 0=no	17%
Year submission 2017	1=yes, 0=no	31%
Year submission 2018	1=yes, 0=no	28%
Year submission 2019	1=yes, 0=no	24%
Total cost major project
Total cost: 50 million euro	1=yes, 0=no	14%
Total cost: 50-100 million euro	1=yes, 0=no	35%
Total cost: 100-200 million euro	1=yes, 0=no	26%
Total cost: 200-400 million euro	1=yes, 0=no	18%
Total cost: > 400 million euro	1=yes, 0=no	7%
EU amount major project
Sector major project
Energy and solid waste	1=yes, 0=no	7%
Rail, air, maritime	1=yes, 0=no	32%
Roads	1=yes, 0=no	37%
Smart development and ICT	1=yes, 0=no	12%
Water and wastewater	1=yes, 0=no	12%

Note: 201 new major projects submitted and with no missing data for all variables.

Table A2.2.2: Ordered logistic regression
Use of Art 102.2 by new major projects	Coef.	z-value	P(z)
Jaspers' TA assistance (1=yes, 0=no)	-0.96	-2.04	0.041
Total cost (relative to > 400 million euro)
Total cost: 50 million euro	-2.18	-2.27	0.023
Total cost: 50-100 million euro	-1.82	-2.05	0.041
Total cost: 100-200 million euro	-1.89	-2.05	0.040
Total cost: 200-400 million euro	-0.94	-1	0.318
Year submission (relative to 2019)
Year submission 2015-2016	2.38	3.3	0.001
Year submission 2017	1.95	3.96	0.000
Year submission 2018	1.16	2.53	0.011
Sector (relative to Rail, air, and maritime)
Energy and solid waste	1.30	1.75	0.081
Roads	0.90	1.81	0.070
Smart development and ICT	1.13	1.9	0.057
Water and wastewater	-0.10	-0.18	0.860
Number projects: 201; LR chi2(12)=47.64; Prob > chi2=0, Pseudo R2=0.196. Statistical significance of 5% if P(z) <=0.05.

M2-3 : Models for the ratio of duration of core preparation in total project duration

(Section 5.2 Effectiveness, ESIF mandate, Advisory major projects)

Table A2.2.3 : Description of variables in the model

Dependent variable	Values	Mean
Ratio preparation in total project duration	0% -100%	65%
Independent variables	Values	Share of "yes"
JASPERS' involvement in core preparation	1=yes, 0=no	60%
JASPERS' from start of core preparation	1=yes, 0=no	12%
JASPERS' involvement after start of core preparation	1=yes, 0=no	48%
JASPERS' involvement after core preparation	1=yes, 0=no	5%
Total costs major project (million euro)
Total cost: 50 million euro	1=yes, 0=no	14%
Total cost: 50-100 million euro	1=yes, 0=no	36%
Total cost: 100-200 million euro	1=yes, 0=no	27%
Total cost: 200-400 million euro	1=yes, 0=no	16%
Total cost: > 400 million euro	1=yes, 0=no	8%
Sector major project
Energy and solid waste	1=yes, 0=no	8%
Rail, air, maritime	1=yes, 0=no	33%
Roads	1=yes, 0=no	40%
Smart development and ICT	1=yes, 0=no	8%
Water and wastewater	1=yes, 0=no	11%

Note : 166 new major projects submitted and with no missing data for all variables.

Source : Based on data from JASPERS’ JADE database and European Commission SFC2014.

Table A2.2.4: Linear regression models with robust standard errors

Ratio core preparation in total project duration	Model 1		Model 2
	Coef.	t-value (P(t))	Coef.	t-value (P(t))
Jaspers' involvement in core preparation	0.06	1.50 (0.136)
Jaspers' involvement from start of core preparation			-0.13	-2.07 (0.040)
Jaspers' involvement after start of core preparation			0.07	1.81 (0.073)
Project total costs (relative to > 400 million euro)
Total cost: 50 million euro	-0.14	-1.85 (0.006)	-0.12	-1.76 (0.081)
Total cost: 50-100 million euro	0.13	2.00 (0.048)	0.14	2.40 (0.018)
Total cost: 100-200 million euro	0.11	1.62 (0.107)	0.11	1.89 (0.061)
Total cost: 200-400 million euro	0.08	1.19 (0.235)	0.08	1.25 (0.214)
Sector (relative to Roads)
Energy and solid waste	-0.39	-7.25 (0.000)	-0.38	-7.21 (0.000)
Rail, air and maritime	-0.29	-6.85 (0.000)	-0.28	-6.82 (0.000)
Smart development and ICT	-0.30	-3.67 (0.000)	-0.24	-3.26 (0.001)
Water and wastewater	-0.29	-4.24 (0.000)	-0.22	-3.55 (0.001)
Country fixed effects	YES
Model fit tests	R-squared = 0.5781; Root MSE= 0.1649		R-squared = 0.6253; Root MSE= 0.15596

Note : 166 new major projects submitted. Statistical significance of 5% if P(z) <=0.05.

M4-M5 : Models for the probability that a project is not interrupted during approval/ adoption procedures

(Section 5.2 Effectiveness, ESIF mandate, Advisory major projects)

Table A2.2.5: Description of variables used in the model

Variable	Values	Share of "yes"
Projects not interrupted (dependent variable)	1=yes, 0=no	43%
JASPERS' early involvement in core preparation	1=yes, 0=no	24%
JASPERS' later involvement in core preparation		32%
Total costs
Total cost: 50 million euro
Total cost: 50-100 million euro	1=yes, 0=no	12%
Total cost: 100-200 million euro	1=yes, 0=no	33%
Total cost: 200-400 million euro	1=yes, 0=no	26%
Total cost: > 400 million euro		20%
Sector major project
Energy and solid waste	1=yes, 0=no	8%
Rail, air, maritime	1=yes, 0=no	27%
Roads	1=yes, 0=no	44%
Smart development and ICT	1=yes, 0=no	12%
Water and wastewater	1=yes, 0=no	8%

Note: 142 major projects submitted for Art 102.2 with no missing data for all variables.

Table A2.2.6 : Logistic regression

Probability the project is not interrupted	Model 1		Model 2
	Coef.	z-value (P(z))	Coef.	z-value (P(z))
JASPERS' early involvement (1=yes, 0=no)	1.04	1.775 (0.076)	2.32	2.909 (0.004)
JASPERS' later involvement (1=yes, 0=no)	0.64	1.295 (0.195)	1.22	2.022 (0.043)
Total cost (relative to > 400 mil. euro)
Total cost: 50 million euro	0.55	0.682 (0.495)	0.09	0.095 (0.924)
Total cost: 50-100 million euro	0.67	0.982 (0.326)	0.54	0.692 (0.489)
Total cost: 100-200 million euro	-0.06	-0.092 (0.927)	0.18	0.231 (0.817)
Total cost: 200-400 million euro	0.24	0.335 (0.738)	0.79	0.960 (0.337)
Sector (relative to Roads)
Energy and solid waste	-1.53	-2.129 (0.033)	-1.80	-2.198 (0.028)
Rail, air and maritime	-0.98	-1.810 (0.070)	-0.44	-0.686 (0.493)
Smart development and ICT	-1.79	-2.551 (0.011)	-0.84	-0.860 (0.390)
Water and wastewater	-1.41	-1.825 (0.068)	-1.87	-2.067 (0.950)
Country fixed effects	NO		YES
Model fit tests	LR chi2(10) = 10.74 Prob > chi2=0.3779 Pseudo R2=0.0554		LR chi2(12) = 38.71 Prob > chi2=0.002 Pseudo R2=0.1995

Note : 142 new major projects for Art 102.2. Statistical significance of 5% if P(z) <=0.05.

M6-M7: Models for financial progress in implementation of approved major projects

(Section 5.2 Effectiveness, ESIF mandate, Advisory major projects)

Table A2.2.7: Variables used in the model:

A) Dependent variable:

Financial progress (share certified expenditure by 2019/2018)

1_No progress 0%	16%
2_Progress up to 10%	12%
3_Progress in 10% -50%	28%
4_Progress in 50%-80%	14%
5_Progress in 80%-100%	20%
6_Progress more than 100%	10%

B) Independent variables :

Variable	Values	Share of "yes"
Jaspers' advisory for core preparation	1=yes, 0=no	60%
Jaspers' advisory early for core preparation	1=yes, 0=no	27%
Jaspers' advisory early for core preparation of young projects (start>=2017)	1=yes, 0=no	19%
EU amount (ERDF and CF)
EU amount: 50 million euro	1=yes, 0=no	50%
EU amount: 50-100 million euro	1=yes, 0=no	25%
EU amount: 100-800 million euro	1=yes, 0=no	25%
Sector major project
Energy and solid waste	1=yes, 0=no	8%
Rail, air, maritime	1=yes, 0=no	33%
Roads	1=yes, 0=no	40%
Smart development and ICT	1=yes, 0=no	8%
Water and wastewater	1=yes, 0=no	11%
Year start project implementation
Year <= 2014	1=yes, 0=no	21%
Year 2015	1=yes, 0=no	20%
Year 2016	1=yes, 0=no	19%
Year 2017	1=yes, 0=no	22%
Year >=2018	1=yes, 0=no	19%

Note : 166 new major projects adopted with no missing data for all variables.

Source : Based on data from JASPERS’ JADE database and European Commission SFC2014.

Table A2.2.8 : Ordered logistic regressions

Note : 166 new major projects adopted. Statistical significance of 5% if P(z) <=0.05.

Annex 3: JASPERS SUPPORT IN 2007-2013

RELEVANCE

Table A3.1 JASPERS completed assignments for ESIF, 2007-2013, by sector and type 170

Source : Jaspers Annual Reports 2010-2013.

Table A3.2 : JASPERS completed assignments for ESIF, 2007-2013, by country and type

Notes : n.a = not applicable.

Source : Jaspers Annual Reports 2010-2013 and European Commission SFC2007, downloaded September 2020.

Table A3.3 : Major projects adopted for the period 2007-2013, by Member State

Country	Number of major projects adopted by EC	Jaspers TA support (% projects)	Framework contract for appraisal (% projects)
(1)	(2)	(3)	(4)
Bulgaria	17	94%	6%
Croatia	11	18%	9%
Cyprus	3	100%	33%
Czechia	45	91%	20%
Estonia	8	50%	25%
France	33	0%	27%
Germany	37	0%	27%
Greece	57	5%	53%
Hungary	39	100%	23%
Ireland	3	0%	33%
Italy	51	14%	90%
Lithuania	5	100%	100%
Latvia	6	100%	33%
Malta	6	100%	33%
Poland	215	60%	42%
Portugal	26	0%	58%
Romania	93	89%	6%
Slovenia	14	93%	14%
Slovakia	27	100%	19%
Spain	60	0%	75%
United Kingdom	11	0%	73%
Total	767	50%	39%

Source: European Commission SFC2007, downloaded September 2020.

Table A3.4 Major projects adopted for the period 2007-2013, by sector

Source: European Commission SFC2007, downloaded September 2020.

Table A3.5 : Median values for project financing by sector and JASPERS advisory

Note : Data covers all 767 major projects approved by EC for the period 2007-2013.

Source: European Commission SFC2007, downloaded September 2020.

Table A3.6 : Topics of JASPERS technical advice for projects supported, 2007-2011

Topic Jaspers' contribution	Projects assisted by Jaspers on the topic	% in all projects assisted
Cost benefit analysis	122	74%
Funding and financing plan	58	35%
Environmental issues	48	29%
Project concept and programming	40	24%
Project implementation and structures	40	24%
Demand analysis	40	24%
Project design	35	21%
Risk and sensitivity analysis	31	19%
Assistance with interruptions	18	11%
Project cost estimation	16	10%
Procurement	16	10%
State aid	13	8%

Note : Shares inferred by authors based on the findings reported. Data cover 165 projects.

Source : European Commission, DG REGIO evaluation of JASPERS 2012.

Table A3.7 JASPERS’s assignments for ESIF non-major projects by sector, 2007-2013

Sector	Number assignments	% in total
Knowledge economy, energy and waste	44	33%
Roads	12	9%
Rail, air and maritime	21	16%
Urban infrastructure and services	6	5%
Water and wastewater	50	38%
Total	133	100%

Source : Jaspers Annual Reports 2010-2013 and European Commission SFC2007, downloaded September 2020.

Table A3.8 JASPERS’s assignments for ESIF non-major projects by country, 2007-2013

Country	Number assignments	% in total
Bulgaria	24	18%
Cyprus	5	4%
Czechia	5	4%
Estonia	2	2%
Hungary	2	2%
Latvia	1	1%
Lithuania	2	2%
Malta	9	7%
Poland	26	20%
Romania	34	26%
Slovakia	5	4%
Slovenia	18	14%
Total	133	100%

Source : Jaspers Annual Reports 2010-2013 and European Commission SFC2007, downloaded September 2020.

EFFECTIVENESS

Table A3.9 Interruption topics for major projects, 2007-2011

Topic	Proportion projects interrupted
Environmental issues	43%
Financial plan	40%
Cost benefit analysis	34%
Project design	32%
Project concept	24%
Implementation and structures	24%
Risk and sensitivity analysis	23%
Project costs	14%
Feasibility study	13%
Procurement	8%
State aid	4%

Note : Data covers 146 major projects.

Source : European Commission, DG REGIO evaluation of JASPERS 2012.

Annex 4: JASPERS SUPPORT 2014-2020

RELEVANCE

A4.1 ESIF Mandate

Table A4.1 JASPERS’ assignments ESIF mandate for project and horizontal support

Sources : JASPERS’s JADE database, December 2018 ; European Commission SFC2014, October 2019.

Table A4.2 JASPERS’ portfolio for review assignments

Sources : JASPERS’s JADE database, December 2018.

Table A4.3 New major projects approved/ adopted by EC until October 2019 171

Country	Art. 102.1 (IQR)		Art 102.2 (PSA)		Total projects	% with JASPERS' TA in total
	Projects	% with JASPERS TA	Projects	% with JASPERS TA
(1)	(2)	(3)	(4)	(5)	(6)=(2)+(4)	(7)
Austria			1	100%	1	100%
Bulgaria	1	100%	9	100%	10	100%
Czechia	3	67%	1	100%	4	75%
Denmark	2	0%			2	0%
Spain			4	75%	4	75%
France	2	100%	3	67%	5	80%
Greece			8	75%	8	75%
Croatia	11	91%			11	91%
Hungary			11	100%	11	100%
Italy	2	100%	5	20%	7	43%
Latvia	3	100%			3	100%
Lithuania	3	100%			3	100%
Malta	1	100%			1	100%
Poland	20	60%	81	44%	101	48%
Portugal			6	33%	6	33%
Romania	5	100%	14	93%	19	95%
Slovenia			4	100%	4	100%
Slovakia	7	100%	3	100%	10	100%
Total	60	80%	150	61%	210	67%

Source : European Commission SFC2014, October 2019.

Table A4.4 JASPERS’ assignments for non-major projects

Country	On-going	Completed and closed	Total	% in total
Bulgaria	3	15	18	13%
Croatia	23	23	46	33%
Cyprus	2	0	2	1%
Czechia	1	2	3	2%
France	0	1	1	1%
Greece	1	3	4	3%
Hungary	0	2	2	1%
Latvia	0	2	2	1%
Malta	1	2	3	2%
Poland	8	5	13	9%
Portugal	1	1	2	1%
Romania	23	17	40	29%
Slovakia	1	1	2	1%
Slovenia	0	2	2	1%
Total	64	76	140	100%
% in total	46%	54%

Sources : JASPERS’s JADE database, December 2018.

Table A4.5 Horizontal and strategic support

Country	Assignments	% in total
Bulgaria	10	8%
Croatia	14	11%
Cyprus	1	1%
Czechia	3	2%
Greece	2	2%
Hungary	5	4%
Ireland	1	1%
Latvua	4	3%
Lithuania	5	4%
Malta	4	3%
Poland	20	16%
Romania	39	31%
Slovakia	11	9%
Slovenia	5	4%
Spain	1	1%
Total	125	100%

Note : Data covers assignments completed and on-going by 2018.

Source : JASPERS’ JADE database, December 2018.

Table A4.6 Interruption rates by country for major projects submitted for Art. 102.2

Country	Assisted by JASPERS TA		Not assisted by JASPERS TA		Overall rate of interruption (%)
	Number projects	% interrupted	Number projects	% interrupted
Austria	1	0%	0	n.a.	0%
Bulgaria	9	67%	0	n.a.	67%
Czechia	1	0%	0	n.a.	0%
France	2	50%	1	100%	67%
Greece	6	67%	2	50%	63%
Hungary	11	73%	0	n.a.	73%
Italy	1	100%	4	100%	100%
Poland	36	25%	45	29%	27%
Portugal	2	0%	4	100%	67%
Romania	13	54%	1	100%	57%
Slovenia	4	100%	0	n.a.	100%
Slovakia	3	33%	0	n.a.	33%
Spain	3	100%	1	100%	100%
Total	92	48%	58	43%	46%

Note : n.a.= not applicable.

Source : European Commission SFC2014, October 2019.

A4.2 CEF1.1 Mandate

Table A4.7 JASPERS’s assignments and projects approved for CEF financing, by country

Country	CEF projects cohesion envelope	CEF projects blending	Projects in Annex I, SGA	Projects supported by JASPERS	JASPERS TA assignments	% JASPERS supported in all CEF
Austria		3
Belgium		2
Bulgaria	6	1	3	3	3	43%
Croatia	17	1	11	2	2	11%
Cyprus	2
Czechia	39	1
Denmark		2
Estonia		1
Finland		4
France		8
Germany		7
Greece	6		2
Hungary	25		19	8	8	32%
Ireland		2
Italy		8
Lithuania	2
Malta	2
Netherlands		7
Poland	28	4
Portugal	9	1
Romania	13		17	5	7	38%
Slovakia	17		5	8	8	47%
Slovenia	6	1	1	1	1	14%
Spain		3
Sweden		4
Multi country	21	12		2	2	6%
Total	193	72	58	29	31	11%

Note : Data covers only projects approved for CEF financing, and considers only the CEF1.1 mandate for JASPERS.

Sources : DG MOVE dataset Autumn 2019, SGA DG MOVE-JASPERS for CEF1.1, Final report for CEF1.1 mandate.

A4.3 IPA Mandate

Table A4.8 : JASPERS’ IPA portfolio by country and type of assignment

Country	Project preparation	Capacity building and horizontal	Implementation	Total	% total
Kosovo		1		1	1%
Montenegro	4	9	4	17	17%
North Macedonia	25	5	1	31	30%
Serbia	28	7	2	37	36%
Turkey	16			16	16%
Total	73	22	7	102	100%
% total	72%	22%	7%	100%

Source : JASPERS JADE database, December 2018.

Table A4.9 Approved IPA projects

Country	Approved projects	Assisted by JASPERS
Albania	32
Bosnia Herzegovina	45
Croatia	13
Kosovo	27
North Macedonia	25	5
Montenegro	33	2
Serbia	47	10
Multi country	31
Total	253	17

Source : DG NEAR dataset, August 2019.

EFFECTIVENESS

A4.4 CEF MANDATE

Table A4.10 Sampled assignments for JASPERS’ advisory for CEF1.1 mandate

Source: JASPERS’ project development reports issued for the assignments.

EFFICIENCY

A4.5 Analysis of country plans for assignments

Table A4.11 Summary results from the analysis of country plans for sampled countries

Planning coordinates	Croatia	Poland	Romania
Assignments matched in JADE and country plans	86%	87%	74%
Assignments in country plans but not JADE	7%	7%	17%
Assignments with consistent data in plans and JADE	79%	86%	86%
Assignments with expected completion date	87%	98%	92%
Assignments on track or completed on time (relative to initial planning)	23%	43%	45%
Median initial delay (for delayed assignments)	19 months	12 months	20 months
Prolongued assignments (updated expected completion date)	78%	90%	86%
Median duration of prolongation	31 months	24 months	32 months

Table A4.12 Administratively closed assignments across countries

Country	All assignments	Administratively closed	% admin closed in country's assignments	% admin closed in total
Austria	1	0	0%	0%
Bulgaria	69	3	4%	3%
Croatia	142	30	21%	31%
Cyprus	5	0	0%	0%
Czechia	38	7	18%	7%
France	12	0	0%	0%
Greece	25	2	8%	2%
Hungary	47	4	9%	4%
Ireland	2	0	0%	0%
Italy	23	1	4%	1%
Kosovo	1	0	e0%	0%
Latvia	13	3	23%	3%
Lithuania	11	1	9%	1%
Malta	13	2	15%	2%
Montenegro	17	4	24%	4%
North Macedonia	31	1	3%	1%
Poland	128	10	8%	10%
Portugal	5	0	0%	0%
Romania	191	13	7%	14%
Serbia	37	7	19%	7%
Slovakia	57	6	11%	6%
Slovenia	16	2	13%	2%
Spain	9	0	0%	0%
Turkey	16	0	0%	0%
United Kingdom	2	0	0%	0%
Multi country	38	0	0%	0%
Total	949	96	10%	100%

Note : Includes assignments for advisory, capacity building, and horizontal and strategic support for all mandates.

Source : JASPERS’ JADE database, December 2018.

COHERENCE

Table A4.13: Comparison of PSA reports and ACNs for projects with non-critical issues in PSA

Topic with non-critical issue(s) in PSA report	Number projects	Conclusions in ACN:			% significant issues and comments
		Significant issue	Assessed, with comment	Assessed, no issues raised
Cost benefit analysis	46	1	19	26	43%
Feasibility study	40	0	27	12	68%
Risk and sensitivity analysis	45	1	21	23	49%
Project description	19	1	0	17	5%
Project costs	21	1	5	14	29%
Capacity of beneficiary	38	4	4	29	21%
Financial plan	35	4	11	25	43%
Project timetable	54	2	11	32	24%

Note: Based on analysis of 77 major projects submitted for Art. 102.2. Data in table reports on the projects which do not have critical issues for the topic, but do have non-critical issues identified in the PSA report.

Source: PSA reports and ACNs.

(1) 2006 MoU signed between the EC, the EIB, the European Bank for Reconstruction and Development (EBRD) and later the Kreditansalt fur Wiederaufbau (KfW), covering the 2007-2013 programming period. For the 2014-2020 programming period, the initiative has been continued on the basis of a framework partnership agreement (FPA) signed between the EC and the EIB.

(2) Albania, Bosnia and Herzegovina, Kosovo*, the Republic of North Macedonia, Montenegro, Serbia, Turkey.* This designation is without prejudice to positions on status, and is in line with UNSCR 1244 and the ICJ Opinion on the Kosovo declaration of independence.

(3) Violeta Piculescu (DG REGIO), 2021.

(4) COWI, Ecorys, CSIL (2021).

(5) https://www.eca.europa.eu/Lists/ECADocuments/SR18_01/SR_JASPERS_EN.pdf

(6) Final version of 6 September 2005.

(7) Cyprus, Czechia, Estonia, Hungary, Latvia, Lithuania, Malta, Poland, Slovakia and Slovenia which joined the EU on 1 May 2004.

(8) Bulgaria and Romania, which joined the EU on 1 January 2007, as of then having access to ERDF and CF financing.

(9) JASPERS (2011), JASPERS Strategy post-2013 needs assessment and preliminary approach.

(10) 14 MS participated in the survey. The full list of analysed sectors was: roads, public transport, water and wastewater, flood risk, climate, solid waste, energy, knowledge economy. More details on the needs analysis performed by JASPERS services are available in Section 4.3.1 of the Final Report of the external study on the Mid-term Evaluation of the JASPERS initiative in 2014-2020.

(11) Referred to as the ESIF (European Structural and Investment Funds) mandate.

(12) Signed in 2015 (CEF 1.1) and 2018 (CEF 1.2). The first CEF mandate had the objective of providing JASPERS’ technical assistance for the preparation of eligible projects, with the intention of being submitted, to CEF calls for the cohesion envelope of EU financing. Countries and projects to be assisted were identified based on a list of priority projects annexed to the first SGA (Annex I) from 2015. The second mandate expanded the geographical scope of assistance to all Member States and focused more on innovative forms of financing (blending). The CEF mandate was initiated in the period of the evaluation as a test pilot initiative with a limited budget.

(13) Signed in 2016, divided into IPA II mandate covering all pre-accession countries for 2016-2021 and the Serbia National Programme Agreement for 2016-2022. For readability, both agreements are referred to in the SWD as the IPA II mandate. The IPA II mandate aimed at providing technical support for all phases of preparatory work for the projects which might be potentially financed via EU funds, as well as capacity building activities.

(14) The JASPERS IQR unit was created in 2014 and was staffed to appraise all categories of major projects developed by the advisory units. The details regarding the activity of Independent Quality Review are explained in the following paragraphs.

(15) In the case of the IPA II mandate, the assignments are undertaken directly by JASPERS, the latter informing DG NEAR when a new assignment has started.

(16) Version 1.4 has been in force since June 2019. The first version was produced in November 2010.

(17) Major projects were defined as projects for which the total eligible costs exceed EUR 50 million, except for the transport sector for which the threshold is EUR 75 million – see Article 100 of the CPR. Non-major projects were projects which do not fulfill the regulatory requirements for a major project. Regulatory requirements for the application for EU funding for a major project were laid down in Article 101 CPR.

(18) Applications for EU financing for a major project were subject to EC approval, which was not requested for non-major projects approved at national level during the 2014-2020 programming period. Applying for EU funding for a major project entailed detailed preparation of project technical specifications, a feasibility study, a cost-benefit analysis and an environmental impact assessment. The project application had to summarise not only the technical, economic and financial feasibility of the project documented in the above studies, but also aspects relating to the alignment of the project with EU cohesion policy objectives, and with the EU requirements and standards regarding environmental impact, adaptation and mitigation measures for climate change, and EU competition rules. The technical advisory services targeting non-major projects were organised in a similar manner. Although the CPR did not mention regulatory requirements for non-major projects, most of the beneficiaries chose to apply a similar approach as the one for major projects (i.e. development of a feasibility study, cost-benefit analysis). In addition, the non-major projects were expected to comply with the EU requirements and standards regarding environmental impact, adaptation and mitigation measures for climate change, and EU competition rules.

(19) The requirements for the independent experts carrying out the quality review and the steps involved in this are set out in Commission Delegated Regulation (EU) 480/2014. The template for the notification of a major project based on Article 102(1) CPR is included in Commission Implementing Regulation (EU) No 1011/2014.

(20) Article 101 CPR lists the documents necessary for the approval of a major project submitted under the procedure based on Article 102(2).

(21) A third procedure based on Article 103 CPR refers to phased projects. These are projects approved in 2007-2013 and which are continued with a distinct phase in 2014-2020. The evaluation focuses on project preparation in 2014-2020; it therefore focuses less on phased projects as they were already prepared in the previous period, and are therefore not comparable with the new major projects.

(22) A critical issue is defined as an instance of not fulfilling the minimum criteria for quality review of major projects listed in the Commission Delegated Regulation (EU) No 480/2014, or of non-compliance with relevant EU legal obligations. A non-critical issue applies when the quality standards for the project, or information requirements, are not fully met but this does not affect the overall positive assessment of the project. Based on JASPERS’ quality manual (2018).

(23) www.jaspersnetwork.org

(24) The EU amount and cumulative investment linked to the project.

(25) The time elapsed between the submission of the major project to the EC and its approval by the EC.

(26) For example, the 2014-2020 legal framework introduced a more rigorous and comprehensive cost-benefit analysis methodology for major projects that defined new elements such as content of option analysis, and climate change proofing - Annex II to Commission Implementing Regulation (EU) No 207/2015.

(27) The European Bank for Reconstruction and Development (EBRD) also provided a small number of experts.

(28) The total number of major projects (assisted and non-assisted for preparation) adopted for the period 2007-2013 is 767, mobilising close to EUR 65 billion from the ERDF and Cohesion Fund. This, however, understates the total amount of EU funding for the projects assisted, as it does not include the support for non-major projects, for which the EC does not have an estimate (non-major projects are approved directly at the level of MAs).

(29) Calculation based on the SGAs. Following decommitments, the EC’s investment was close to EUR 140 million.

(30) The European Bank for Reconstruction and Development (EBRD) also provided a small number of experts.

(31) As noted above, the EU resources mobilised by ESIF projects assisted by JASPERS advisory are understated, as they do not include the EU funding for the non-major projects assisted and approved at national level, for which the EC does not have an estimate of the total funding involved. The EC had approved 210 new major projects by the end of October 2019.

(32) Administratively closed assignments are assignments that were closed before their objective could be achieved. The timeline and reasons for administrative closure of assignments are analysed in detail in Section 5, in the sub-section on efficiency.

(33) As explained, many of these assignments were started prior to 2014, in preparation for the period.

(34) The CEF1.2 mandate started in 2018, so there is only a few months of overlap with the period covered by this evaluation and, accordingly, limited scope to assess it.

(35) An additional three projects supported during CEF1.1 were postponed for CEF1.2 and are so not included in the CEF1.1 mandate.

(36) Of these, the budget execution rate is highest for the Serbia mandate (75% as compared with 59% for IPA II).

(37) The analysis and findings for the effectiveness, efficiency and EU added-value criteria were mostly supported by circumstantial evidence or statements of a qualitative nature collected from stakeholders.

(38) A proxy analysis aims to provide an indirect measure of relevant outcomes. It is commonly used when direct measures of outcomes are unobservable and/or unavailable.

(39) This analysis covers all 150 major new projects submitted to the Commission under the submission procedure (with PSA review) during the reference period (whether assisted or not), with a focus on the projects interrupted in the approval process in the Commission. The approval process of a major project is interrupted when the Commission identifies critical issues in the project documentation. If this happens, an observation letter is sent to the Member State signalling the issues. The process is resumed once the reply of the Member State is received.

(40) The most common reasons for interruption related to significant issues which are: cost-benefit analysis, feasibility studies, state aid, risk and sensitivity analysis, and capacity of the beneficiary. The analysis covers 37 projects assisted by JASPERS advisory, interrupted by the Commission, and for which the complete JASPERS support and project documentation was available. In total, 69 major projects were interrupted or withdrawn until October 2019.

(41) 142 major projects were included in the analysis (with complete data sets of the necessary variables), out of 150 major projects adopted after the submission procedure (with PSA review).

(42) Starting with the entry into force of the Omnibus Regulation (EU, Euratom) 2018/1046 in July 2018, a regulatory deadline of 6 months was introduced for the IQR assessment.

(43) JASPERS services were not involved in the approval process of major projects in the previous programming period of 2007-2013.

(44) However, because of a lack of data it was not possible to establish whether those participating in capacity-building events were also the ones working on project preparation. In addition, the requirements for major projects were somewhat different between the two programming periods, so the expertise gained by the Member States in the past required updating. More information on the requirements linked to major projects in the 2007-2013 period can be found in the JASPERS evaluation report (AECOM, 2012).

(45) The number of assignments undertaken (29) was directly dependant on the budget available, which was limited since the initiative was at a pilot stage. The combined estimated total costs of the projects linked to the JASPERS assignments supported though CEF1.1 mandate amounts to over EUR 8 billion.

(46) Two of these projects were approved for IPA financing, with the remaining 4 either not being submitted (2 projects) or not being approved due to cuts in IPA financing.

(47) https://www.eca.europa.eu/Lists/ECADocuments/SR18_01/SR_JASPERS_EN.pdf

(48) The assessments for Croatia and Poland were carried out in cooperation with the national supreme audit institutions.

(49) Only starting from 2017, as this is when JASPERS began recording the number of hours worked per assignment.

(50) More than half of the administratively closed assignments were in Croatia, Poland, and Romania.

(51) Data available starting from 2017.

(52) The PSA assignments had a clear deadline of 30 days agreed between JASPERS and the Commission. This deadline was introduced since the Commission itself is bound by regulatory deadlines for issuing a decision on major projects submitted, and therefore there was a need for predictability as regards time delivery of PSA reports.

(53) In relation to the IQR function, there was no regulatory deadline up until July 2018, and therefore the process tended to be prolonged for some assignments. This is also because JASPERS IQR division needs to undertake an interactive process of assessment with Member States. This happens, for instance, because of the need to clarify certain aspects of projects with Member States during the assessment. In the future, however, it is expected the IQR process will become more limited in time because of the regulatory deadline of 6 months introduced in the Omnibus Regulation in 2018.

(54) A more detailed analysis of this aspect of efficiency is not feasible at present, as it would require data on hours worked per assignment and calendar year for all assignments – data which are not available.

(55) The templates for submitting information on major projects for the ERDF and Cohesion Fund are included in Commission Implementing Regulation (EU) 2015/207 Annex II (for the procedure based on Article 102(2)), and in Commission Implementing Regulation (EU) 1011/2014 Annex II (for notification based on Article 102(1)).

(56) Regulation (EU) 2015/1017.

(57) Valid for ESIF and CEF mandate, for the IPA mandate it is only informing the Commission.

(58) The consultation took place at a moment when the 2021-2027 cohesion policy package was still under negotiation. It therefore cannot be ruled out that there was a certain attempt by Member States to influence its outcome.

(59) Throughout the evaluation a distinction is made between JASPERS’ assignments for project preparation and the projects themselves. This is due to the fact that it is possible that one assignment covers more than one projects, or that a project can be covered by more than one assignments. We explain in text whenever this is the case.

(60) This was the latest possible date when we could update the data based on the SFC2014 system before consolidating the analysis for the evaluation.

(61) See Annex 2 of this Staff Working Document.

(62) Among the sampled assignments, 77% are for the ESIF mandate, and 11% the CEF and IPA mandate each, covering all JASPERS’ services. JASPERS provided all available documentation for the sampled assignments including, for instance, project fiches, ACNs, guidance notes, value added fiches, and feedback data from beneficiaries.

(63) See Annex 2 of this Staff Working Document.

(64) Until 2013, two more partners were involved in setting up the initiative: the European Bank for Reconstruction and Development (EBRD) and Kreditansalt fur Wiederaufbau (KfW).

(65) CEF is the EU funding instrument supporting interconnected trans-European networks in the field of transport, energy, and digital services. JASPERS’ CEF mandate in the period 2014-2020 covers CEF projects in the field of transport.

(66) The IPA II instrument sets the framework for pre-accession assistance for 2014-2020. Beneficiary countries include Albania, Bosnia and Herzegovina, Kosovo, Montenegro, North Macedonia, Serbia, and Turkey.

(67) Framework Partnership Agreement (FPA) for the management of the JASPERS technical assistance facility 2014-2020. The FPA is based on the broader Financial and Administrative Framework Agreement (FAFA) signed by the European Commission and the EIB in 2014.

(68) JASPERS IQR unit was created in 2014 and was staffed to appraise all categories of major projects developed by the advisory units (based on JASPERS’ Quality Manual 2018). The reader should note that all legislative texts and documents cited in text are presented will full details in the last section of References.

(69) Major projects are defined as projects for which the total eligible costs exceed EUR 50 million, except for the transport sector for which the threshold is EUR 75 million. For more details see the Common Provisions Regulation (EU) No 1303/2013, Art. 100.

(70) Regulatory requirements for the application for EU funding for a major project are laid down in the Common Provisions Regulation (CPR) (EU) 1303/2013, Art. 101. Regulation (EU) 2015/207 includes the template for the application in Annex II, and the minimum requirements for a cost benefit analysis in Annex III. Furthermore, for the cost benefit analysis, the European Commission also published a guide to CBA of investment projects for Cohesion Policy in 2014 (see DG REGIO(2014) in References).

(71) Non-major projects are projects which do not fulfill the regulatory requirements for a major project explained earlier.

(72) The Common Provisions Regulations (CPR) stands for Regulation (EU) No 1303/2013.

(73) The requirements for the independent experts carrying out the quality review and the steps for the quality review are regulated in Commission Delegated Regulation 480/2014. The template for the notification of a major project based on CPR Art. 102.1 is included in the Commission Implementing Regulation (EU) No 1011/2014.

(74) CPR Art. 101 lists the documentation necessary for the approval of a major project submitted for the procedure based on Art. 102.2.

(75) A third procedure based on CPR Art. 103 refers to phased projects. These are projects approved during 2007-2013, and which are continued with a distinct phase in 2014-2020. The evaluation focuses less on phased projects since, from the perspective of project preparation in 2014-2020, they are already prepared in the previous period, and therefore not comparable with the new major projects.

(76) A critical issue is defined as an instance of not fulfilling the minimum criteria for quality review of major projects listed in the Commission Delegated Regulation (EU) No 480/2014, or of non-compliance with relevant EU legal obligations. A non-critical issue applies when the quality standards for the project, or information requirements, are not fully met but this does not affect the overall positive assessment of the project (based on JASPERS’ Quality Manual (2018)).

(77) More details on the structure of these reports are included in the section for coherence in Chapter 5.

(78) The CEF cohesion envelope is the budget transferred from the Cohesion Fund to the CEF funding programme. This budget is reserved for projects in Member States eligible for the Cohesion Fund. Cohesion Member States include: Bulgaria, Croatia, Cyprus, Czechia, Estonia, Greece, Hungary, Latvia, Lithuania, Malta, Poland, Portugal, Romania, Slovakia, and Slovenia.

(79) The European Investment Advisory Hub (EIAH) is a partnership between the EIB and the Commission as part of the Investment Plan for Europe. It is designed to act as a single access point to various services for advisory and technical assistance. It supports the identification, preparation and development of investment projects across the European Union.

(80) CEF Blending Operations include investments combining the use of grants and/ or financial instruments from the EU budget and financing from Implementing partners (if under CEF Blending Facility) or from financing institutions (private or public for 2017 Blending calls) (via loan, debt, equity or any other repayable form of support).

(81) Albania, Bosnia and Herzegovina, Kosovo, the Republic of North Macedonia, Montenegro, Serbia and Turkey.

(82) See JASPERS’ Quality Manual (2018) in References.

(83) The total number of major projects (assisted and non-assisted for preparation) adopted for the period 2007-2013 is 767, mobilising close to 65 billion euro from ERDF and Cohesion Fund. The EU resources mobilized by projects assisted by JASPERS mentioned in text, however, underestimate the total amount of EU funding for assisted projects as they do not include the support for non-major projects, for which we do not have an estimate.

(84) See EIB (2010) in References.

(85) See DG REGIO (2012) in References.

(86) See ECA (2018) in References.

(87) The assessments for Croatia and Poland were carried out in cooperation with the national supreme audit institutions.

(88) The EU resources mobilized by ESIF projects assisted by JASPERS advisory are underestimated as they do not include the EU funding for the non-major projects assisted by JASPERS and approved by Managing Authorities. We do not have an estimate for the total funding approved for such projects.

(89) Administratively closed assignments are assignments which were closed before the objective of the assignment could be achieved. The timeline and reasons for administrative closure of assignments are analysed in details in Chapter 5, in the section for efficiency.

(90) As explained in text, many of these assignments were started prior to 2014, in preparation for the period.

(91) Throughout the evaluation we make a distinction between JASPERS’ assignments for project preparation and the projects themselves. This is due to the fact that it is possible that one assignment covers more than one projects, or that a project can be covered by more than one assignments. We explain in text whenever this is the case.

(92) This was the latest possible date when we could update the data based on the SFC2014 system before consolidating the analysis for the evaluation.

(93) Summaries of the replies to the consultation are included in Annex 1. The report with full details of the results from the targeted consultation is published on the website of this evaluation.

(94) Among the sampled assignments, 77% are for the ESIF mandate, and 11% the CEF and IPA mandate each, covering all JASPERS’ services. JASPERS provided all available documentation for the sampled assignments including, for instance, project fiches, ACNs, guidance notes, value added fiches, and feedback data from beneficiaries.

(95) For further details on these models, see Long and Freese (2013) in References.

(96) At a later stage in the evaluation process, we learnt that JASPERS formalized also the IQR process starting primarily with year 2019, which is outside the main reference period for this evaluation.

(97) As mentioned in the first chapter, the cutoff date for the reference period for the evaluation of JASPERS’ portfolio of assignments is 31 December 2018.

(98) As matching the phased projects to advisory assignments in JASPERS’ portfolio proved more difficult, the rate of JASPERS assistance for these projects may be slightly underestimated.

(99) This data includes only two projects assisted by JASPERS advisory for implementation.

(100) For phased projects, the project documentation had already been prepared and approved during the previous programming period and the application for funding in the current programming needs to provide sufficient assurance that the phased project has not changed significantly and it still complies with the regulatory requirements.

(101) Please note that the data for 2007-2013 covers the full period and therefore is not perfectly comparable with the first 5 years of 2014-2020. Nevertheless, we consider that this comparison can still be relevant for emerging trends.

(102) In Poland, the majority of the non-assisted major projects for Roads are prepared according to the “Blue Book” developed with the support of JASPERS advisory during 2007-2013. This illustrates the potential for JASPERS’ technical advisory to have a long lasting impact on technical capacity at country level.

(103) For Water and wastewater the comparison cannot be made since all projects in this sector were assisted by JASPERS advisory for preparation.

(104) There are 92 major projects submitted for Art. 102.2 and which had been supported by JASPERS advisory for preparation. Of these projects, Member States provided the ACNs for 81 projects together with project application. We note that there is no regulatory obligation for the Member States to provide the ACNs of a project assisted by JASPERS together with the project application, although we consider it as a good practice. By comparison, for projects submitted for adoption based on Art 102.1, the corresponding ACNs (if applicable) are not available in the Commission’s SFC2014 system. Starting with 2019, however, JASPERS sends all the ACNs for the completed assignments to the Commission, and these are subsequently distributed to the relevant desk officers in DG REGIO.

(105) See the section on effectiveness for a more detailed classification.

(106) We present the timeline of project preparation and implications for JASPERS’ work in more details in the analysis of effectiveness in the next section.

(107) It may also be an indication that the manner in which the Member States approach JASPERS’ advisory assistance was not fully coordinated.

(108) Italy, a Member State also sampled for this evaluation and relevant for the ESIF mandate, had no assignments for non-major projects until end 2018.

(109) The ESIF portfolio includes 66 assignments completed for non-major projects.

(110) The difference between median and average costs in this care is not high. The average total cost is EUR 36 million for the 39 ESIF projects with data available. This does not take into account one outlier of EUR 484 million which refers to an assignment for project initially conceived as a major project.

(111) We also found that, in some cases, projects were initially prepared as major projects and reclassified as non-major at later stages.

(112) Regulation (EU, Euratom) 2018/1046.

(113) It is also possible that Member States were initially reluctant to use the procedure based on Art 102.1 since it was very new compared with the more established procedure of Commission’s assessment.

(114) The estimates for sector have a weaker statistical significance.

(115) Some country specific HSS assignments for workshops and seminars are not included here but they are taken into account in the classification for capacity building.

(116) www.jaspersnetwork.org

(117) Additional 3 projects supported during CEF1.1 were postponed for CEF1.2. Therefore, we do not include them in the CEF1.1 mandate.

(118) The budget of EUR 1.5 million allocated to the first pilot for the CEF1.1 mandate was first decided by the Steering Committee of CEF, and subsequently allocated by country according to pre-determined national allocation keys.

(119) And 5 of these assignments for studies were still ongoing by end 2018.

(120) As one of these assignments was administratively closed, the relevant document reviewed was the action closure note.

(121) Projects are also included for Croatia for the period before its accession to the EU.

(122) We took projects starting in 2011 since the first JASPERS’ assignment for project preparation in the portfolio for 2014-2020 is for a project starting in 2011.

(123) Additional 3 projects for which the advisory assignment had been administratively closed were also subsequently adopted by the Commission until October 2019. Overall, there are 160 projects which had been assisted by JASPERS advisory and were subsequently adopted/ approved by the Commission until October 2019.

(124) These stages include: 1. feasibility study/ business plan, 2. cost-benefit analysis, 3. environmental impact assessment, 4. design studies, 5. preparation of tender documentation, 6. tender procedures, 7. land acquisition, 8. development consent, 9. construction phase, 10. operational phase. For project implementation (the operational phase), the timetables include expected completion dates.

(125) There are 139 assignments for advisory for new major projects which were subsequently submitted to the Commission until October 2019. For this analysis we used all projects with complete data both in JADE and in SFC.

(126) These are projects for which we could find data for project timetables and all co-founding factors.

(127) Projects submitted for Art 102.1 are not included since the IQR assessment process is more informal and has no comparable interruption procedure.

(128) During the current period, there has also been a significant reorganization of activities overseeing the approval of major projects in DG REGIO, notably with the creation of a dedicated horizontal unit coordinating these activities at the service level.

(129) Also, the regulatory requirements for major projects were further developed in the current period, relative to 2007-2013.

(130) These proxy measures are, however, imperfect as the observations included in interruption letters refer to both critical and non-critical issues with project quality.

(131) ACNs provided by Member States together with the application for major projects for Art. 102(2) are available in SFC2014 for 88% of all projects assisted by JASPERS advisory over the reference period.

(132) These projects represent 93% of all 178 new major projects adopted/ approved by the Commission until October 2019.

(133) For physical progress in implementation none of the three variables for JASPERS advisory are statistically significant. Therefore, the corresponding model results are not reported.

(134) We found also that, according to JADE, they started earlier than the project preparation specified in the project timetable. This can happen, for instance, if the project was assisted also for the previous programming period.

(135) Starting with the Omnibus Regulation adopted in July 2018, the regulatory requirement is for the IQR assessment to be completed within 6 months.

(136) The period of 3 months is the regulatory deadline for tacit approval. Quicker approval is possible only in exceptional situations.

(137) Average duration for Art 102(2) is 4.5 months.

(138) Average duration for Art 102(1) is 7.8 months.

(139) We do not distinguish between critical and non-critical issues in the overall assessment since Commission’s observations usually cover both types of issues.

(140) JASPERS’ Quality Manual (2018).

(141) An informal argument among stakeholders consulted suggested that capacity building is better strengthened with hands-on experience while working directly with JASPERS on the projects rather than through trainings and workshops. While, in principle, we see the merit of this argument, we also do not see any evidence that this is reflected in actual sustainably better preparation of projects across sectors and countries. This may be due to higher turnover of the personal in national authorities, but also due to the fact that often JASPERS interacts with the consultants contracted for the preparation of project documentation and therefore the learning effect for the staff of national authorities is limited.

(142) Due to lack of data, however, we could not really establish whether the persons who participated in capacity building events were also the ones working on project preparation.

(143) Republic of Croatia State Audit Office (2017) – Impact of the JASPERS programme on submitted project proposals by the Republic of Croatia for co-financing from EU funds. Report of Performance Audit, Zagreb, May 2017.

(144) The issue of administratively closed assignments is further analysed in the next section on efficiency.

(145) For two of these projects we also found that JASPERS had assisted them in the past for project preparation when they were being planned for the ESIF mandate.

(146) For the IPA mandate, one assignment can cover several projects, and one project can be covered by more than one assignments.

(147) Most of the assignments matched with approved IPA projects appear with the status “ongoing” in JADE by end December 2018.

(148) There are 36 assignments completed and administratively closed for advisory for project preparation for the IPA mandate. Therefore, the sampled assignments represent 17% of the total.

(149) The 6th assignment was administratively closed as the beneficiary did not make any progress with project preparation.

(150) Country plans do not include the assignments for the review function (IQR and PSA) for ESIF.

(151) Availability across countries differs slightly. For example, for Poland, the earliest country plan available was from April 2016. The analysis, however, also takes into account the coverage of the time period by the plans.

(152) For a number of early assignments we could not establish whether they were present both in country plans and in JADE due to availability of country plans. For Croatia, for example, the country plans available for analysis start in the 2nd quarter of 2015. Therefore, for all assignments created and completed in JADE earlier than this period we could not determine whether they we present also in country plans. The analysis, however, takes into account this aspect.

(153) Normally, once an assignment is completed or closed, it is no longer included in subsequent country plans.

(154) 94% of the assignments in this dataset are for ESIF, 4% for IPA, and 2% for CEF.

(155) This assumption is based on the fact that a calendar year of 365 days usually includes 250-255 working days. Therefore we made a round approximation of 70% working days within a calendar year.

(156) In addition to reported working hours per assignment, JASPERS estimates that the time needed for ex-ante preparation of assignments and other horizontal tasks amounts to 12% of the overall working time.

(157) Intensity of work can be zero for assignments administratively closed, and higher than 100% if more experts work at the same time on the assignment.

(158) For the CEF1.2 mandate, the allocated EC contribution is EUR 3 million.

(159) Of these, the budget execution rate is higher for the Serbia mandate (75%), compared to IPA II (59%).

(160) Directive 2011/92/EU.

(161) Directive 92/43/EEC.

(162) Additional possible reasons include the omission of statements in the IQR reports or statements rendered unreliable due to changes that have occurred in the project between the finalization of the IQR and the notification to the Commission.

(163) A list of these initiatives is presented in ECA (2020).

(164) Regulation (EU) 2015/1017.

(165) This is likely to change in the future as, for the next MFF, the Commission proposed the creation of the InvestEU Advisory – a centrally managed initiative which will include also JASPERS.

(166) The online consultation includes also a probing question, whereby stakeholders are asked about their familiarity and use of other advisory services such as the Structural Reform Support Service of the Commission, other EIB advisory services, the World Bank advisory, EBRD advisory, and private consultancy services. A large of majority of 90% of respondents stated that they had used at least one of these advisory services, and therefore the replies to the following questions are likely to be informed by their direct experience. Table A1.12 in Annex 1 presents the details.

(167) In the value added fiche, these aspects are classified as follows: increased capacity of counterparts, contribution to sector strategy/ planning, contribution to removal of barriers to realise projects, cost savings, improvements of project quality, facilitation of project implementation, and other.

(168) This estimate applies to major projects submitted for Art.102.2 and assisted by JASPERS advisory.

(169) With this occasion, we would like to express our gratitude to all stakeholders in Member States and pre-accession countries, and to our Commission colleagues, who cooperated for this evaluation.

(170) The classification of assignments by sector differs slightly from the current period since the codes used have changed over time.

(171) SFC2014 dataset includes additional projects for which the review/ assessment was carried out in 2019 (therefore, after the reference end date for the JADE dataset).

Term or acronym	Meaning or definition
CEF	Connecting Europe Facility
CF	Cohesion Fund
CPR	Common Provisions Regulation, Regulation (EU) 1303/2013
DG MOVE	Directorate General for Mobility and Transport
DG NEAR	Directorate General for European Neighbourhood Policy and Enlargement Negotiations
DG REGIO	Directorate General of Regional and Urban Policy
EC	European Commission
EIB	European Investment Bank
ERDF	European Regional Development Fund
ICT	Information and communication technology
IPA	Instrument for Pre-accession Assistance Fund
IQR	Independent quality review
JASPERS	Joint Assistance to Support Projects in European Regions
MoU	Memorandum of Understanding
MS	Member State
PSA	Post-submission appraisal
SGA	Specific Grant Agreement
SWD	Staff Working Document

Choose the experimental features you want to try