EUROPEAN COMMISSION

Brussels, 26.1.2026

SWD(2026) 980 final

COMMISSION STAFF WORKING DOCUMENT

EVALUATION

of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726

Accompanying the document

Report from the Commission to the European Parliament and the Council

on the evaluation of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726

{COM(2026) 32 final}

Table of Contents

1.Introduction

1.1.    Purpose and scope of the evaluation    

2.What was the expected outcome of the intervention?

2.1    Description of the intervention and its objectives    

2.2    Points of comparison    

3.How has the situation evolved over the evaluation period?

4.Evaluation findings

4.1.    To what extent was the intervention successful and why?    

4.1.1.    Effectiveness    

4.1.2.    Efficiency    

4.1.3.    Coherence    

4.2.    How did the EU intervention make a difference and to whom?    

4.3.    Is the intervention still relevant?    

5.What are the conclusions and lessons learned?

5.1.    Conclusions    

5.2.    Lessons learned    

Annex I: Procedural Information

Annex II. Methodology and Analytical models used

Annex III. Evaluation matrix

Annex IV. Overview of benefits and costs

Annex V. Stakeholders consultation - Synopsis report

Glossary

1.1.Purpose and scope of the evaluation

The European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) was established by means of Regulation (EU) No 1077/2011 1 with a view to ensuring the operational management of existing large-scale IT systems in the area of freedom, security and justice (at that stage: the second generation Schengen Information System – SIS II, the Visa Information System – VIS, and Eurodac) and potentially to support the development and operation of additional IT systems to be established. The establishing Regulation was amended on 20 July 2015 by Regulation (EU) No 603/2013 (Eurodac recast Regulation) 2 to reflect the changes brought by this Regulation with regard to the Agency's tasks related to Eurodac.

Following an evaluation of the Regulation in 2017 3 , the mandate of eu-LISA was revised based on Regulation 2018/1726 (hereinafter referred to as “eu-LISA Regulation”), the subject of this evaluation. While Regulation 2011/1726 was fully replaced and repealed by eu-LISA Regulation, the new Regulation builds largely upon the previously established legal framework. The general objective of the Agency thus remains to “contribute to a high-level security within the area of freedom, security and justice, including by facilitating border management and law enforcement”. In addition, the Agency has the following specific objectives based on eu-LISA Regulation:

·To contribute to a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice that is in line with fundamental rights;

·To contribute to the development of IT-based solutions in the area of freedom, security and justice; and

·To contribute within the scope of its mandate to evidence-based Union migration and security policy-making and to the monitoring of the proper functioning of IT systems in the area of freedom, security and justice.

Article 39 4 of the eu-LISA Regulation calls for an evaluation of the performance of the Agency in relation to its objectives, mandate, locations and tasks with a view of identifying potential areas for improvements. The Commission is also required to report to the European Parliament (EP), the Council and the Management Board (MB) of eu-LISA on the findings of the evaluation.

The Commission carried out the supporting study between October 2023 and November 2024, covering the period from the entry into force of the current eu-LISA Regulation in 2018 until March 2024. 5 The evaluation also compared the situation with the period between 2012 and 2018 before the current regulation entered into force. While the study formally concluded in 2024, the evaluation integrates developments up to June 2025. Extending the evaluation period allowed the Commission to reflect the significant organisational and management changes that the Agency underwent in parallel to the study, thereby ensuring that the assessment captures the most recent progress and provides a more comprehensive and accurate picture of the Agency’s performance.

As regards methodology, a mixed methods approach has been used, including information gathered via desk research, surveys, interviews and field visits, and was informed by the triangulation of a variety of sources. A range of methodological tools and techniques were used. For more details on the methodology please see Annex II.

As part of this evaluation, a wide range of stakeholders were consulted through various methods (interviews, surveys, workshops, etc.). The stakeholders consulted include the European Commission, staff and contractors of eu-LISA, Member States authorities participating in the governance bodies of eu-LISA (including Advisory Groups, Project Management Boards and Management Board members) from both the Home Affairs and Justice domains, EU Agencies and Offices (including the European Border and Coast Guard Agency, Eurojust and European Public Prosecutor’s Office), the European Data Protection Supervisor (EDPS) and national data protection authorities. A detailed description of the stakeholder consultation is presented in Annex V.

This evaluation analyses whether the eu-LISA Regulation contributed to the achievement of the general objectives of the Agency in an effective, efficient and coherent manner. This evaluation includes an analysis of the legal and policy frameworks governing eu-LISA and focus largely on the performance of the tasks, the organisation set-up, including the governance and locations of the Agency, and the management of economic, human and technical resources.

Furthermore, the evaluation analyses the relevance and EU added value of the Agency, by assessing to what extent the new mandate of the Agency, introduced in 2018, has contributed to achieving the objectives of the Agency, and to which extent it has supported Member States in the development and implementation of the large-scale IT systems in the Area of Freedom, Security and Justice. Additionally, the evaluation analyses whether the objectives of eu-LISA could have been achieved sufficiently by Member States acting alone.

Finally, the evaluation looks at the relevance of eu-LISA in the context of current and emerging needs and challenges. This includes assessing the relevance of the eu-LISA regulation in terms of its overall scope and objectives.

The evaluation takes stock of the state of play in the implementation of the eu-LISA Regulation and identifies a series of areas of attention for improvement that will feed into the future work of the Commission, eu-LISA and Member States.

2.1Description of the intervention and its objectives

Objective of eu-LISA

The establishment of a European Agency for the operational management of large-scale IT systems in the area of freedom, security and justice (“eu-LISA”) dates back to the adoption of its Regulation in 2011 as a solution for the long-term operational management of SIS II, VIS and EURODAC. In 2017, the Commission published its first evaluation of the initial eu-LISA Regulation. 6 Albeit the evaluation concludes that eu-LISA overall effectively ensured the management of large-scale IT systems and created clear added value, it found significant room for improvement.

Following the 2017 evaluation, the Commission adopted a new eu-LISA Regulation, Regulation (EU) 2018/1726, 7 which is the subject of this evaluation. The objective was to address the recommendations esteeming from the evaluation, as well as to improve the functioning of the Agency and to enhance and strengthen its role to ensure that the mandate met the challenges at EU level in the area of freedom, security and justice. This new instrument, aimed also at inserting in the Regulation changes deriving from policy, legal or factual developments and in particular to reflect the fact that new systems were be entrusted to the Agency and the responsibility of the Agency to develop the interoperability between the large-scale IT systems. In particular, these changes included:

-Giving eu-LISA enhanced responsibilities with regard to data quality subject to the adoption of specific legislative amendments/proposals.

-Giving eu-LISA responsibility for the development of interoperability actions subject to the adoption of the relevant legislative proposals.

-Addressing the possible need for eu-LISA to develop, manage and/or host joint technical solutions for the national implementation of technical aspects of obligations deriving from EU legislation on decentralised systems in the area of freedom, security and justice for interested Member States; and

-Addressing changes required by the adoption of the ECRIS-TCN proposal.

This new instrument extended the mandate of the Agency to allow it to take over new tasks in order to render border management more effective and secure and to reinforcing security and combatting and preventing crime. The general objective of the Agency thus remains to “contribute to a high-level security within the area of freedom, security and justice, including by facilitating border management and law enforcement”. In addition, the Agency has the following specific objectives based on the establishing eu-LISA Regulation, and the amendments introduced by the IT systems’ establishing regulations (cf. the intervention logic, see Figure 1):

-To contribute to a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice that is in line with fundamental rights.

-To contribute to the development of IT-based solutions in the area of freedom, security and justice, including the takeover of e-CODEX; and

-To contribute within the scope of its mandate to the implementation of EU policies and to the monitoring of the proper functioning of IT systems in the area of freedom, security and justice.

The evaluation

Article 39 of the eu-LISA Regulation calls for an evaluation of the performance of the Agency in relation to its objectives, mandate, locations and tasks with a view to identifying potential areas for improvement.

The strategic objective of this evaluation was to assess whether, based on its Regulation 8 , eu-LISA functions smoothly and with high added value to ensure the effective and efficient implementation of EU policies in the area of cross-border law enforcement, border management, and justice. It aimed to examine how eu-LISA Regulation contributes to a coordinated, cost-effective and coherent IT environment with a view to identifying potential areas for improvement (cf. Article 39).

As regard the subject matter scope, the evaluation examined the implementation of eu-LISA Regulation, covering all objectives and tasks of eu-LISA based on the Regulation as well as aspects relating to its organisational set-up, management and oversight. As a result, the evaluation developed a series of recommendations based on the findings relating to the previous points and overarching conclusions and recommendations on the performance and functioning of the Regulation.

In terms of the scope from a time perspective, the evaluation concerns the period after the entry into force of eu-LISA Regulation, i.e. November 2018 until March 2024, but also compares the situation against the time before it entered into force based on the previous evaluation 9 . While the evaluation is retrospective, recommendations for improvement have also been developed. The geographical scope covers the EU-level, EU Member States and Schengen Associated Countries (“Member States”).

Intervention logic

For the purpose of the evaluation, a detailed analysis was carried out identifying the Agency’s objectives (general and specific) and tasks and the links between them. The intervention logic below (Figure 1) serves to depict the chain of expected effects associated with the eu-LISA Regulation, by identifying the needs and objectives of eu-LISA, as well as inputs and actions to achieve the outputs, outcomes, and impacts in line with the five evaluation criteria (relevance, coherence, effectiveness, efficiency and EU added value).

The intervention logic shows the needs leading to the intervention logic, namely the smooth operation of large-scale IT systems to respond to challenges relating to border management and law enforcement within the Schengen area.

In particular, the intervention logic depicts the objectives of eu-LISA at different levels:

·One general objective: to contribute to a high level of security within the area of freedom, security and justice, including by facilitating border management and law enforcement within the area of freedom, security and justice.

·Three specific objectives:

oTo contribute to a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice that is in line with fundamental rights;

oTo contribute to the development of IT based solutions in the area of freedom, security and justice; and

oTo contribute to evidence-based Union migration and security policy-making and to the monitoring of the proper functioning of IT systems in the area of freedom, security and justice; and

·Fourteen operational objectives.

The intervention logic identified the inputs, including human, financial and technical resources, structures and processes dedicated to achieving the operational, specific and ultimately general objectives of the Agency.

Finally, the intervention logic translates the objective and inputs into a hierarchy of expected effects, including at the level of outputs, outcomes and impacts.

The effectiveness is measured by how the objectives (general, specific and operational) are translated into expected outputs, expected outcomes, and ultimately expected effects. The efficiency compares the inputs with expected outputs, expected outcomes, and expected effects. The coherence measures to what extent the Regulation avoid gaps and overlaps and assesses the degree of internal coherence as well as the extent to which it is align with other relevant EU legal instruments and policies. The relevance measures to which extent the objectives (general, specific and operational) identified to achieve the needs translate into the expected outputs, outcomes and impacts. Finally, the EU added value measures how to which extent the EU intervention helped to achieve the defined objectives.

Figure 1: Intervention logic of eu-LISA’s Regulation

2.2    Points of comparison 

In 2017, pursuant to Article 31 of the eu-LISA establishing Regulation (EU) No 1077/2011 10 , the European Commission carried out an evaluation of the Agency (hereinafter the 2017 evaluation) to:

·Examine the way and the extent to which the Agency effectively contributes to the operational management of large-scale IT systems in the area of freedom, security and justice and fulfils its statutory tasks; and

·Assess the role of the Agency in the context of a Union strategy aimed at a coordinated, cost-effective and coherent IT environment at Union level.

Given that no formal Union strategy for a coordinated, cost-effective and coherent IT environment had been put in place at the time, the evaluation focused on the Agency’s contribution to the establishment of a coordinated, effective and coherent IT environment for managing large-scale IT systems that support the implementation of Justice and Home Affairs (JHA) policies.

The 2017 evaluation examined the way and the extent to which the Agency effectively contributes to the operational management of large-scale IT systems in the area of freedom, security and justice and the implementation of eu-LISA's tasks taking into account the relevant legal and policy frameworks, namely the eu-LISA Agency establishing Regulation and the legislative instruments governing the IT systems the Agency manages. The 2017 evaluation covered the period from 1 December 2012 to 30 September 2015.

The 2017 evaluation identified two general objectives: (i) to ensure the operational management of SIS II, VIS and Eurodac; and (ii) to ensure the preparation development and operational management of other large-scale IT systems in the ASFJ. In addition, seven specific objectives, considered the expected results of the Agency, are identified: (i) to ensure effective, security and continuous operation of large-scale IT systems; (ii) to ensure efficient and financially accountable management of large-scale IT systems; (iii) to ensure an adequately high quality of service for users of large-scale IT systems; (iv) to ensure continuity and uninterrupted services; (v) to provide a high level of data protection; (vi) to provide an appropriate level of data and physical security; and (vii) to use an adequate project management structure for efficiently developing large-scale IT systems. Finally, the 2017 evaluation identified sixteen tasks of eu-LISA and analyses how the Regulation has contributed, through the different tasks, to the achievement of the objectives.

The 2017 evaluation confirmed that the Agency effectively ensures the operational management of large-scale IT systems in the area of justice and home affairs and fulfils the tasks laid down in the establishing eu-LISA Regulation as well as new tasks entrusted to it. It also found that eu-LISA effectively contributed to the establishment of a more coordinated, effective and coherent IT environment for the management of large-scale IT systems that support the implementation of Justice and Home Affairs (JHA) policies. The 2017 evaluation therefore provided the necessary reassurance about the continuing validity of the rationale behind creating eu-LISA and its further existence. In particular:

·In regard to effectiveness: the 2017 evaluation found that the Agency had overall achieved its objective of effectively ensuring the operational management of the three IT systems entrusted to it.

·In regard to efficiency: the 2017 evaluation found that there is a need for a full review of the allocation of resources for core and non-core tasks of eu-LISA, to be performed by the Agency, in order to ascertain whether the allocation is proportionate to the aims to be achieved and whether some reallocation could occur (for instance the development of new systems). The Evaluation also found that there was a need to put appropriate controls in place to ensure that sufficient staff resources are available for project management based on project needs.

·In terms of coherence: the 2017 evaluation found that the activities of eu-LISA are generally well-aligned with the relevant JHA policy frameworks and strategies and the coherence of the Agency’s activities was confirmed.

·Finally, in terms of relevance and added value: the 2017 evaluation found that the establishment of a single management authority to assume operational management of the three IT systems inherently creates a high level of added value, to the extent that the Agency effectively and efficiently carries out its tasks. However, ultimate added value of eu-LISA is thus logically highly dependent on its ability to fulfil its core tasks in an efficient and effective manner.

Nevertheless, the 2017 evaluation also identified certain shortcomings for all evaluation criteria at both operational and organisational levels and recommended a number of changes to address them. Although the Agency proved that it could do more with the same level of resources and can adapt with a high degree of flexibility to new needs, the 2017 evaluation also concluded that, should eu-LISA be made responsible for new IT systems, it would not be able to manage them with the resources available. In its actions, the Agency could be more efficient, effective and coherent and could also strengthen its relevance and added value.

The 2017 evaluation came up with a set of recommendations for legislative amendments to the eu-LISA establishing Regulation and to other related instruments, which included:

·The transfer of the European Commission’s responsibilities relating to the communication infrastructure to eu-LISA.

·A new provision on the cooperation framework of eu-LISA with other JHA agencies to clarify the scope of cooperation within eu-LISA’s mandate.

·An extension of the scope of the pilot schemes that can be entrusted to eu-LISA.

·Additional tasks in relation to eu-LISA management of the systems, including: (i) an extended responsibility for eu-LISA in generating statistical reports for each system; and (ii) a new task to produce data quality and data analysis reports.

Finally, the evaluation identified the possible need to increase the tasks entrusted to eu-LISA and introduced a limited set of recommendations for legislative amendments to the eu-LISA establishing Regulation and to other related instruments, including changes derives from amended EU legislation (such as the Financial Regulation and the Framework Financial Regulation), changes derived from the adoption by the co-legislators of the Commission proposals that entrust new systems to the Agency such as the Entry/Exit System, and changes to allow eu-LISA to provide assistance or support to the relevant Commission services on technical issues related to existing or new systems, where requested.

The same day of the adoption of the 2017 evaluation report, the Commission tabled, on the basis of the evaluation recommendations, a proposal to amend the eu-LISA establishing Regulation and the system’s instruments. eu-LISA’s mandate was revised based on Regulation (EU) 2018/1726 (the “eu-LISA Regulation”), and the role in the development of new large-scale IT systems considerably expanded, as further detailed in Section 3.

Considering the responsibilities of the Agency, the 2018 eu-LISA Regulation established a series of tasks that were latter extended through amendments of the eu-LISA Regulation by the IT systems establishing Regulations 11 . The tasks of the Agency detailed in the eu-LISA Regulation (article 3 to 16) are the following:

·Adding responsibilities relating to new IT systems and Interoperability (Article 6, 7, 8, 8a, 8b, 8c, 13):

oThe Entry/Exit System (EES), the European Travel Information and Authorisation System (ETIAS), DubliNet, and the interoperability of IT systems were included in the 2018 eu-LISA Regulation;

oThe computerised system for the cross-border electronic exchange of data in the area of judicial cooperation in civil and criminal matters (e-CODEX), the European Criminal Records Information System for third-country nationals (ECRIS-TCN), the European Criminal Records Information System Reference Implementation, and the joint investigation teams collaboration platform (JITs CP) were added by later amendments in 2019, 2022 and 2023 triggered through the systems’ establishing Regulations);

·Detailing of the tasks relating to the operation of IT systems, including requiring eu-LISA to develop solutions to develop technical solutions and meet the requirements for systems which are running 24 hours a day, 7 days a week, ensuring data quality and interoperability (Article 10, 12, 13);

·Specification of tasks relating to the communication infrastructure (Article 11);

·Providing a possibility for eu-LISA to support the Member States and the Commission on request (Article 16);

·Enhancing the Agency’s responsibilities in research and pilot testing (Article 15); and

·Inclusion of a possibility for eu-LISA to collaborate with international organisations, only with the authorisation of the Management Board after having received the Commission’s prior approval (Article 43).

A summary of the tasks of eu-LISA can found in Figure 2 below.

Figure 2: Overview of eu-LISA's taks

It should be noted that additional tasks have been added after the start of this evaluation, in particular in regards to API-PNR (Advanced Passenger Information – Passenger Name Record), as well as in relation to the tasks related to the Prüm II router (Article 8d) 12 . These tasks are, however, not in scope of this evaluation.

The governance structure of eu-LISA has also evolved since 2018, consisting of the following bodies:

·The Management Board (MB) is made up of representatives from the EU Member States and the European Commission, represented by DG HOME (Article 20). In addition, Schengen Associated Countries (SAC) are Members of the Management Board 13 , while Eurojust, Europol, Frontex and the European Public Prosecutor’s Office (EPPO) have observers status. The role of the MB is to provide the overall orientation for eu-LISA’s activities and oversee the implementation of eu-LISA’s tasks (Article 19). The Management Board also appoints the Executive Director, who is accountable to the MB (Article 25). The MB meets ca. five times annually. To focus on specific business-related aspects, the MB created an Audit Compliance and Financial Committee (ACFC) p, an informal sub-group of the MB The Management Board adopted the ACFC’s first mandate on 20-21March 2018. The ACFC held its first meeting on 19 September 2018. The mandate of the ACFC was revised by Management Board Decisions on 20 September 2022 and 15-16 November 2023). 

·The Advisory Groups (AGs): the objective of the AGs is to provide the MB with the necessary technical expertise relating to the IT systems managed by eu-LISA. The AGs are made up of representatives from them EU Member States, SAC and the Commission, as well as in some cases representatives from Europol, Eurojust, Frontex and EPPO as observers.

The eu-LISA Regulation established 4 AGs (Article 27): SIS II, VIS, Eurodac and EES-ETIAS AGs. The number of AGs extended since 2018 to a total of 9, including: the ECRIS-TCN AG (set up in 2019), the Interoperability AG (2019).

·Five Programme Management Boards (PMBs) on ETIAS 14 , EES 15 , ECRIS-TCN 16 , e-CODEX 17 and JITs CP 18 have been established by the establishing Regulations of the relevant IT systems. They comprise seven Member State representatives, the Commission and the chair of the AG of the same system. In some cases, additional eu-LISA representatives may join the PMB (e.g. in the case of the EES and ETIAS PMBs). PMBs advise the MB on aspects relating to the development and/or handover of the related systems.

While the e-CODEX PMB is established as a permanent body (Article 13.1 of the e-CODEX Regulation 19 ), the other PMBs are established on a temporary basis 20 during the designing and development phase of the systems. In addition, the e-CODEX PMB has an additional responsibility to monitor the respect for the principle of the independence of the judiciary and prompt preventive or corrective action where necessary.

Within eu-LISA, the MB Secretariat plays a central role in supporting the governance bodies and ensuring their smooth functioning, including by organising and documenting meetings, ensuring the implementation of decisions taken and communicating with members of different bodies. 21

Figure 3: Governance structure of eu-LISA

During this evaluation, the implementation of eu-LISA’s tasks was organised through three separate departments including 10 units and 29 sectors. In addition, there were in total seven executive units directly connected to the Executive Director. This also comprised the security unit which is, therefore, set up independently from any internal or operational departments. During the Evaluation period, the Agency underwent a revision of the organisational set up.

The Agency’s seat is located in Tallinn, Estonia, while two operational sites are situated in Strasbourg, France. Whereas the Executive Director, the corporate governance and corporate services departments, as well as the security unit are located in Tallinn, the operations department is based in Strasbourg 22 . A third site, which operates as a backup for the systems, is located in St. Johann im Pongau, Austria. The Agency also has a liaison office in Brussels. Most of eu-LISA’s staff is located in Strasbourg (73%). 26% of the Agency’s staff is located in Tallinn and 1% in Brussels.

4.1.To what extent was the intervention successful and why? 

The different evaluation criteria are analysed in the section below.

4.1.1.Effectiveness

This evaluation criterion examined the extent to which eu-LISA has fulfilled its expectations and met its objectives. The evaluation assessed to which extent eu-LISA has contributed to a high level of security within the area of freedom, security and justice, and in particular, eu-LISA’s contribution a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice, eu-LISA’s contribution to the development of IT-based solutions in the area of freedom, security and justice, and eu-LISA contribution within the scope of the mandate to evidence-based Union migration and security policy-making and to the monitoring of the proper functioning of IT systems in the area of freedom, security and justice.

To what extent are procedures in place to ensure the coherence of eu-LISA’s cooperation activities with the policies and activities of its stakeholders?

As outlined in the 2018 eu-LISA Regulation, the core objectives of the Agency embody the following: 23

-The development of large-scale IT systems using an adequate project management structure for efficiently developing such systems;

-The effective, secure and continuous operation of large-scale IT systems;

-The efficient and financially accountable management of large-scale IT systems;

-An adequately high quality of service for users of large-scale IT systems;

-Continuity and uninterrupted service;

-A high level of data protection, in accordance with Union data protection law, including specific provisions for each large-scale IT system; and

-An appropriate level of data and physical security, in accordance with the applicable rules, including specific provisions for each large-scale IT system.

Based on the available evidence, the above-mentioned core objectives are met to a moderate- to large extent. In regard to safeguarding effective, secure and continuous operation; the continuity of uninterrupted services; the provision of data protection; the provision of high-quality services to users; and ensuring data and physical security, the Agency has met its objective to a large extent. This is backed by the available evidence, both from interviews and from the online surveys. In fact, 24 out of 35 (69%) AG and PMB members who answered this specific question indicated in the online survey that eu-LISA’s management of the systems contributes to a ‘large’ or ‘very large’ extent to a coordinated, cost-effective and coherent IT environment at EU level in the area of freedom, security and justice.

As concerns the development of large-scale IT systems as well as the efficient and financially accountable management of IT systems, eu-LISA has met its objectives to a moderate extent due to delays in development processes and lack of accountability.

As concerns the tasks of eu-LISA, we found that system-specific tasks, as well as providing support to Member States are working well. However, according to interviewed stakeholders, there is still room for improvement on communication even within the governance bodies to facilitate more efficient decision-making processes. Nearly half of the respondents of the AG and PMB survey acknowledged that Member States are supported to a “very large” or “large extent”. However, 18% of the respondents did not have a clear opinion on this matter which could indicate a need for eu-LISA to further improve communication and engagement with certain stakeholders to ensure that the benefits of its mandate are fully understood and utilised.

Stakeholder interviews reinforce the perception that eu-LISA is capable of meeting its core objectives. At the same time, stakeholders have been raised concerns that the broadened scope of the latest Regulation could risk generating a higher workload without a corresponding increase in staff. Thee LFS are prepared by the Commission in cooperation with eu-LISA.

Stakeholders interviewed also pointed out concerns about internal administrative processes within the Agency that delay the effective allocation of staff. This suggests communication issues, as well as an internal organisation that does not evolve in line with the growing responsibilities of the Agency.

How has been the performance of eu-LISA in operationally managing large-scale IT systems?

For the purpose of addressing this question on operational management performance, it is important to assess several indicators, including systems availability, number of operational incidents, response to incidents, as well as user satisfaction with data quality or the overall IT infrastructure provided.

Based on available evidence, the Agency is performing well in the context of operational management of its IT systems with some existing areas for improvement.

The perception of eu-LISA stakeholders on the Agency’s performance in operationally managing large-scale IT systems has been largely positive. The Agency has demonstrated a strong capability in handling technical issues, with AG and PMB members expressing satisfaction with response times, incident management, resolution times, and the prevention of operational disruptions.

Most respondents also recognised eu-LISA's contribution to creating a coordinated, cost-effective, and coherent IT environment within the EU's Area of Security and Justice (62 out of 74, 84%). From the perspectives of users of the IT systems this satisfaction is reflected, too. The management of DubliNet, Eurodac, SIS, and VIS is generally seen as satisfactory, with high marks for system availability and operational support. The overall level of satisfaction among AG and PMB members is high: 86% of respondents indicated that they are satisfied or very satisfied.

However, the feedback also shows that the levels of satisfaction vary between different areas: while the level of satisfaction is highest concerning security-related aspects (93% were ‘satisfied’ or ‘very satisfied’), the general level of satisfaction is slightly lower concerning both the Agency’s services (86% were ‘satisfied’ or ‘very satisfied’) and operational requirements for IT systems (77% were ‘satisfied’ or ‘very satisfied’). Only a few users have experienced issues such as temporary system unavailability and data entry errors. .

For users of the IT systems, the efficiency of maintenance work (37 out of 43, 86%), technical development (28 out of 40, 70%), and system availability (39 out of 43, 91%) has been well-received, indicating that eu-LISA is able to ensure its systems to be reliable and up to date.130 However, there are mixed perceptions on the cost-efficiency of communication tools, suggesting a need for eu-LISA to evaluate and possibly optimise the tools used for interacting with Member States.

Despite these successes, stakeholders have identified areas for improvement, such as the use of management systems, tracking of change requests, and communication regarding production issues and test environments. In particular:

-VIS: Limited interoperability, limited scalability of system (due to its age), limited data quality, and limited search functionalities;

-SIS: Limited capacity of eu-LISA to provide further development and improvements, burdensome administrative processes, limited interoperability with other databases and systems, limited data quantity (i.e. not all information that is available in a particular Member State), limited data quality (e.g. the system contains a very high number of alerts on issued documents without any significant practical effect), limited file sizes, limited extent of automation, downtimes, and limited awareness among users concerning specific (new) types of alerts.

-Eurodac: Limited functionalities, limited information exchange after biometric correspondence, and limited ability to triangulate with data in case of false positives. There is room for improvement concerning reaction times.

At the same time, however, available data from the survey with AG and PMB members shows that the Agency has been responsive to feedback and suggestions for improvement of the systems, at least to a moderate or large extent. 24  

To what extent has eu-LISA been effective in ensuring the secure and continuous operational management of large-scale IT systems entrusted to it?

Overall, eu-LISA has been very effective in ensuring the secure and continuous operational management of large-scale IT systems during the evaluation period.

The IT systems currently under operation and management of eu-LISA (in particular SIS and VIS) have so far worked without any major incidents. They have proven to be highly reliable and capable of delivering uninterrupted operations, effectively handling a significant volume of requests 25 26 27 28 . The takeover process of existing IT systems (e.g. e-CODEX) was considered to be efficient to a very large or large extent.

Moreover, the evaluation highlighted the high level of user satisfaction regarding several dimensions, such as security, data protection, and service continuity. In particular, the evaluation underlined the service reliability and efficient security measures of the operating systems. However, the evaluation pointed to differences in the (high) levels of satisfaction between VIS, SIS, and Eurodac and less satisfied with eu-LISA’s performance in relation to Eurodac.

Nevertheless, the evaluation also unveils areas for improvement. For example, response timeliness to incidents and service disruptions could be improved, with the evaluation indicating a need for enhanced agility and responsiveness. For example, for Eurodac, certain reported downtimes were, in fact, rather attributed to the national interfaces. the evaluation also pointed to the lack of information about the origins of outages, likely again attributed to Member States. For the case of the SIS recast, the evaluation highlighted the slow responsiveness of the Agency to questions, as well as the unavailability of the test or production environment at certain uncalculatable times. The proposed 'software factory' model, employing DevSecOps (Development, Security, Operations) principles, underscored the Agency's efforts to bolstering control and reducing dependencies on external contractors.

To what extent has eu-LISA fulfilled its obligation in relation with data protection, data and physical security in accordance with the applicable rules, including specific provisions for each large-scale IT system?

Overall, eu-LISA has fulfilled its obligations reliably concerning data protection, data, and physical security in alignment with applicable rules and specific provisions for each large-scale IT system.

The Agency's proactive approach to data protection is evidenced by several strengths. Notably, eu-LISA has invested in comprehensive training and awareness initiatives to sensitise personnel, ensuring a culture of compliance with data protection principles 30 . Additionally, the establishment of clear storage limitation rules, categorised into Category 1 and Category 2 data, aligns with GDPR principles, ensuring purpose limitation and data minimisation 31 . The involvement of the Data Protection Officer (DPO) in initiatives such as Change Management reflects a robust commitment to data protection by design and by default 32 . Monthly updates of the Record of Processing Activities (RoPA) 33 and frequent monitoring of the data breach register further underscore eu-LISA's commitment to compliance and proactive risk management.

However, the Agency faces significant challenges in the ever-evolving landscape of digital threats and global instabilities. The surge in cybercrime poses a substantial risk, necessitating continuous adaptation to ensure the security of large-scale IT systems 34 . Additionally, the lack of a clear categorisation of incidents, for example, whether or not, they affect system availability poses challenges in implementing adequate mitigating measures promptly 35 .

Furthermore, the workload on the Data Protection Office raises concerns about the effectiveness of their involvement in various initiatives, highlighting a need for internal resource allocation improvements and a clearer definition of priorities. Ensuring that the DPO is adequately resourced and able to focus on the most pressing issues is essential for allowing them to support the Agency in identifying and addressing challenges, thereby reinforcing the Agency’s role in developing the systems. This workload is only expected to increase in the future and technological solutions should be considered according to interviewees. The rapid advancement of disruptive technologies such as AI presents both opportunities and risks, requiring a delicate balance between innovation and data protection 36 . For example, AI can support the DP by automating routine tasks, identifying potential data privacy issues through advanced analytics, and enhancing compliance efforts through real-time monitoring and predictive analysis.

Despite these challenges, stakeholders generally perceive eu-LISA's work on cybersecurity and data protection positively. The Agency's management of communication infrastructure using TESTA-ng ensures secure data transfer, supporting effective decision-making by national authorities 37 .

In conclusion, while eu-LISA has made significant strides in upholding data protection and security obligations, there is always a need for continued vigilance and adaptation to address emerging challenges effectively.

How has eu-LISA performed in the development of new large-scale IT systems?

eu-LISA's performance in developing new large-scale IT systems shows a mixed picture, characterised by successes, but more increasing and pressing challenges with the development of selected new systems such as the EES.

Notably, while users expressed satisfaction with the efficiency of maintenance work, uninterrupted availability, and responsiveness of eu-LISA in managing existing systems, sentiments by survey respondents and interviewees regarding the development of new systems were less favourable. 38

Delays in the development of the EES, but also ETIAS and ECRIS-TCN, have eroded stakeholders' confidence in eu-LISA's capacity to deliver efficiently in the future as can be seen from consultations with the PMB, the user survey as well as concluded from several interviews from both the operational and management perspective. In relation to these systems, significant quality and delivery issues have been encountered which, in turn, resulted in cascading delays that impacted dependent work streams and systems For example, the revised roadmap at the time of the supporting study indicated that EES was planned to commence operations in late 2024, while ETIAS was scheduled to become operational in the first half of 2025. According to the current planning EES is scheduled in late 2025 and ETIAS in late 2026. It should be noted, however, that developing a completely new system and revising or transferring an existing one, such as the recast of the VIS, present distinct challenges. While new system development offers possibilities for flexibility and innovation, it requires extensive planning and testing to ensure compatibility and functionality. Recasting an existing system involves modifying existing infrastructure, which can expedite deployment, but may result in complexities in integrating with legacy systems and accommodating evolving requirements. Whereas eu-LISA was less effective with regard to the development of completely new systems, there are good practices and positive lessons learned from the recast of existing systems, such as the SIS.

Moreover, there have been challenges in taking up new justice-related systems, where functional knowledge had to be built up at eu-LISA such as ECRIS-TCN and the JITs collaboration platform, underscore the complexity of the Agency's mandate. In particular, the development and takeover of justice systems such as e-CODEX has also been highlighted as challenging by stakeholders from the justice area, including members of the e-CODEX AG and PMB. This stems from the fact these IT systems are very different in their functionality and (business) processes from the other systems (such as SIS) that eu-LISA has been managing before. Hence, the necessary awareness and expertise (e.g. regarding the respect of the principle of the independence of the judiciary) had to be gradually built up in order to ensure that no infringements (e.g. at the technical level) take place. On the positive side, the structured and regular participation of members from the e-CODEX community, including those represented in the PMB, has proven beneficial. According to several interviewees from the justice community and in particular the e-CODEX AG and PMB, the extensive experience with and maturity of e-CODEX when it was handed over have, overall, facilitated a smooth transition. According to their perception, eu-LISA has a strong track record of bringing together various communities and has gained substantial knowledge about e-CODEX over the year during the handover. Most stakeholders also see various opportunities for further involvement in the management of justice systems within the Agency, such as the possibility for eu-LISA to contribute with high-level technical expertise in the comitology, as it is now the case in the Home Affairs systems 39 , to become a recognised institution to align technical approaches between Member States and to become a technical pillar of the justice community. At the current stage, however, it was highlighted that the expertise within the Agency is rather technical/law enforcement-related, still requiring additional expertise in the justice sector before it can participate in a role as described above.

While some specific interoperability systems such as the European Search Portal and Shared Biometric Matching Service are perceived as moderately to largely developed by survey respondents, others such as the Multiple Identity Detector exhibit limited progress. Consequently, the Agency's ability to contribute to the timely development of new systems is, on average, rated as only moderately successful by respondents.

A significant concern highlighted by stakeholders is the over-reliance on external contractors for system development. This dependency raises issues of knowledge retention and control over development processes, potentially compromising flexibility and adaptability in the long term. Additionally, resource constraints and shifting priorities contribute to discrepancies between planned and actual activities, posing further challenges to effective system development. Despite these challenges, eu-LISA's introduction of interoperability components such as the CSI programme shows that the Agency is actively addressing historical inefficiencies and enhancing interoperability. The CSI platform aims to streamline operations and improve system scalability, aligning with the Agency's transformative objectives.

In conclusion, while eu-LISA has been effective in managing existing systems and some new interoperability components, the development of new large-scale IT systems presents distinct challenges that require proactive strategies and collaboration with stakeholders. Overall, this reflects a need for strategic adjustments and a concerted effort to overcome existing challenges.

To what extent have eu-LISA tripartite set-up and governance structure affected the performance of the Agency in delivering its core tasks?

This question concerns the split-up of eu-LISA over three main locations in Tallinn (Estonia), Strasbourg (France) and St. Johann im Pongau (Austria), and how this set-up affects the performance of the Agency in delivering its core tasks, which are defined as development and operation of the large-scale IT-systems. For this purpose, it is important to (i) analyse the separation of tasks between the three locations and (ii) analyse survey and interview perceptions on the tripartite set-up.

Based on literature analysed 40 and stakeholders consulted, it became apparent the headquarters in Tallinn are hosting more horizontal and managerial tasks, whereas the two operational sites in Strasbourg are concerned with the actual development and operation of the IT-systems. St. Johann im Pongau serves also as an operational set-up, providing a back-up in case the systems in France should not be usable.

In general, most staff show understanding for the separation of locations in Tallinn and Strasbourg, particularly as concerns the political dimension of the Agency. In terms of the operational dimension, given that the different main areas of work are clearly separated between different locations, daily tasks can be usually organised and completed within a single location, reducing the need for administrative coordination between the different locations. In case of a need for cross-location coordination, existing communication channels and methods seem to be sufficient to meet the requirements of eu-LISA staff. The back-up site in Austria is at the current stage only marginally included in coordination activities with the other two sites, allowing for it to perform its tasks effectively by itself. These findings are also underlined by the AG/PMB survey, with roughly 75% of stakeholders pointing out that the tripartite set-up does only to a “moderate” “low” or “no” extent have an effect on the Agency’s efficiency.

However, some stakeholders from within eu-LISA consulted via interviews have pointed out that the geographical separation of locations can occasionally hinder mutual understanding between staff regarding the concrete and operational needs of other sites.. The physical distance between the sites may also create technical difficulties, such as ensuring availability, connectivity, latency, maintenance, and regular back-ups of systems between the two technical centres in Strasbourg and St. Johann in Pngau.

In addition, stakeholders have highlighted that both main locations come with location-specific advantages and disadvantages, leading to particular challenges at the different locations. In particular Tallinn was portrayed as a major obstacle for recruiting, given the comparably low salaries (for IT talent) as well as the attractiveness of Tallinn as a city (remote location, harsh weather conditions, connectivity). The office premises, however, were portrayed as central, inviting and modern. Strasbourg’s premises, on the other hand, have been described as rather outdated and uninviting, whereas the geographical location of the city is considered an asset. To ensure that staff feel equally valued across eu-LISA’s main locations, further adaptations and investments should be considered to enhance parity in working conditions, for instance in terms of facilities, infrastructure and overall working environment. In the context of a highly competitive IT labour market, where working conditions are key to attracting and retaining skilled professionals, ensuring balanced conditions across sites would help strengthen cohesion, and support staff retention within the Agency.

Nonetheless and only with few exceptions, stakeholders largely agreed that the current set-up as foreseen within eu-LISA’s mandate is overall effective and that no major adaptions are needed within the mandate. Adaptions are rather recommended on the operational and organisational levels, through e.g. increased coordination and teaming efforts.

To what extent have eu-LISA’s management structures and monitoring of external contractors been effective to assure compliance and timely implementation of contracts related to the development and operational management of large-scale IT systems?

This question concerns the effectiveness of eu-LISA’s outsourcing approach in particular with regard to management structures and monitoring capabilities of the Agency. For this reason, it is important to analyse not only how the current management structures look like but also how stakeholders assess the performance of the current structures. In the second step, an assessment of the appropriateness of the structures follows.

As concerns eu-LISA’s management structures, eu-LISA has long prolonged a rather system silo-centric outsourcing approach, in which tasks have been outsourced based on a system-centric model.. 41 To overcome this approach, the Agency has established the more horizontal TEF/TOF (Transversal Engineering Framework / Transversal Operation Framework’) outsourcing framework, in which different types of services (related to e.g. operations, infrastructure, application, testing, development, maintenance, etc.) can be provided by different consortia of contractors for each system. Monitoring is conducted by different actors as e.g. the Finance and Budget Sector, the Vendor and Contract Management Sector or the Business Relations Management Sector. Given the split over different departments, it should be noted that responsibilities are sometimes not clearly distributed, which can lead to inefficiencies in the monitoring process.

As concerns the assessment of stakeholders, most stakeholders confirmed that the outsourcing of certain tasks is necessary and unavoidable, and that eu-LISA is following a collaborative approach, working closely with the external contractors on the implementation of existing and planned systems. Nonetheless, the performance of the management structures is considered rather poor. Many stakeholders consider the TEF/TOF model to be too complex, given the different layers of services and still silo-centric focus on single IT-systems. In fact, the framework is so complex that a specific framework contract has been signed with a contractor to manage the delivery in other framework contracts.

In addition, the general outsourcing strategy was described as insufficient, which is mirrored by the survey results: more than half of respondents (53%, n=39) consider the current strategy “not at all”, to “a low extent” or to a “moderate extent” successful in delivering new systems. In addition, the following weaknesses could be identified based on the data available:

-The overall procurement process is described as rather lengthy and administrative, often not pointing out the requirements of the procurement adequately in its full complexity. Moreover, the fixed price and scope of the contracts leave too little flexibility to undertake adaptions needed due to the dynamic developments in fields relevant for the Agency-managed IT-systems (e.g. through technological advancements such as in the field of AI or new EU and national policies in relevant areas);

-It was underlined frequently that the administrative burden of managing the external contracts is very high, as concerns documentation obligations;

-Most stakeholders pointed out that through the complexity of the contracts managed, a lack of clearly allocated responsibilities leads to a delusion of responsibilities, strongly affecting the effectiveness of administrative processes and slowing down operations;

-Contractors of the Agency noted that eu-LISA’s current mandate is rather restrictive in terms of using new technologies, such as e.g. cloud technology. Therefore, contractors must fall back on more complex and sometimes outdated technologies. Problems with the current contract system, in particular as concerns the development of new systems, is highlighted by the survey. 

Overall, the management and monitoring of external contractors must be concluded to be rather insufficient, which is also underlined through the number of delays and poor quality of work of external contractors, as is highlighted by most stakeholders consulted. Even though there are positive developments since the arrival of the new Executive Director (investment into co-sourcing strategies, working under ‘one roof’ at the same premises), there seems to be significant room for improvement.

To what extent has eu-LISA been effective in ensuring the efficient and financially accountable management of large-scale IT systems? To what extent has eu-LISA been effective in managing its budget: could budgets expenditures be related to objectives/expenditures, are accounts compliant with EU budget rules, are discrepancies between actuals and budgets explained?

eu-LISA has demonstrated an overall effective management of its large-scale IT systems, maintaining a high level of financial accountability.

The Agency has faced an evolving landscape of challenges, such as complex multi-annual planning and rigorous compliance requirements that have necessitated a micro-management approach to budgeting. These challenges have led to some difficulties, particularly in the flexibility and strategic clarity of outsourcing strategies and agile development practices. Issues with outsourcing strategies, mainly the complexity and strategic approach, have particularly affected the delivery of key IT systems, like the EES.

In examining the relationship between budget expenditures and objectives, eu-LISA has put in place mechanisms to align spending with strategic goals. However, the specificity and strictness of legal bases have restricted the organisation’s agility in budget utilisation, making it challenging to swiftly adapt to changing technological and political demands. Despite these constraints, actual expenditures generally correspond to outlined objectives, as supported by eu-LISA's high budget implementation rates.

Regarding compliance with EU budget rules, eu-LISA follows a structured budgetary process. The Agency submits its budget requests, negotiates the amount with the European Commission and eu-LISA's management, and then adheres to EU Financial Regulations. Interviewees have expressed concerns about the restrictive nature of this process and the limited involvement of Member States and the Management Board during negotiations. While this might suggest a need for greater transparency, there is no evidence that eu-LISA has failed to comply with EU budget rules. For 2022, the budget accounts provide a comprehensive picture of the budget implementation, and the outstanding budgetary commitments not yet expensed represent the correction of the budgetary RAL (reste à liquider) with estimated costs determined by applying the accrual-based principle. This indicates a structured approach to relating budgets and expenditures to objectives 42 .

Discrepancies between actuals and budgets are an inevitable part of financial management, particularly in a rapidly changing IT environment. eu-LISA's financial reports indicate that discrepancies are closely monitored and generally explained by the Agency's adjustments to unforeseen changes and challenges. For instance, interviews highlight the need for improved multi-annual planning capabilities and more flexible budgeting practices to handle such discrepancies proactively.

In 2022, eu-LISA effectively managed significant budget appropriations, with commitment appropriations having a 99.0% implementation rate and payment appropriations a 91.6% rate by year-end. These figures attest to the Agency's financial diligence and ability to achieve defined objectives within established financial frameworks. Financial KPIs are consistently met, further showcasing eu-LISA's commitment to financial accountability 43 .

In conclusion, while eu-LISA has faced challenges in refining its financial planning and outsourcing practices, it remains effective in disbursing its budget, ensuring expenditures reflect objectives, and providing accountability for any variances. The Agency's high budget implementation rates, alongside ongoing efforts to enhance budget-related processes and planning, indicates efficiency and financial accountability.

How effective has eu-LISA been in ensuring high quality, continuous and uninterrupted services for users of large-scale IT systems? How effective has eu-LISA been in meeting the Service Level Agreement(s), in particular for availability of service, for each large-scale IT system?

eu-LISA has demonstrated a high level of effectiveness in ensuring the quality and continuity of services provided by its large-scale IT systems, while also fulfilling its obligations under SLAs in terms of service availability. Evidence from desk research and several interviews suggests that eu-LISA has implemented measures to maintain and enhance the data quality across its systems. The Agency has utilised advanced technologies and procedures to ensure accuracy, consistency, and timeliness of the data processed and stored. Regular audits, quality control mechanisms, and feedback loops with Member States contribute to the high data standards eu-LISA upholds. Concrete metrics, such as error rates and correction times, have reportedly been within acceptable thresholds, though specific figures would provide a clearer picture of the performance in this area. 44

Further evidence (e.g. from the user survey) shows that the Agency has managed to provide continuous and uninterrupted services through robust infrastructure and redundancy mechanisms designed to minimise downtime. This has resulted in high system availability and reliability for end-users. As the reports from operational metrics indicate the system uptimes have consistently met or exceeded the targets set forth in the respective SLAs, although exact uptime percentages would offer more substantive evidence.

For each IT system managed by eu-LISA, the achievement of SLA objectives, particularly regarding availability of service, is as follows:

·DubliNet: The system has maintained operational continuity with few reported incidents, meeting its SLA target for availability. Downtime has been minimised through effective incident response strategies 45 ;

·Eurodac: Eurodac has shown high availability rates, aligning with SLA requirements. eu-LISA has addressed technical challenges promptly to ensure the system remains accessible to users 46 47 ;

·SIS (Schengen Information System): SIS has upheld excellent service availability, as eu-LISA has implemented comprehensive monitoring and rapid issue resolution to comply with SLA standards 48 49 ; and

·VIS (Visa Information System): VIS has similarly maintained high availability, with eu-LISA conducting regular maintenance and upgrades to prevent service disruptions in line with SLA commitments 50 51 .

Despite these achievements, eu-LISA continues to face challenges, such as ensuring consistent data quality and addressing interoperability requirements. The development and integration of new systems such as EES,ETIAS and ECRIS-TCN have experienced delays, but eu-LISA's is actively addressing these issues with task forces and managerial escalation (e.g. through the ED). Continuous efforts to update and refine operational processes, systems enhancements, and stakeholder engagement will be key to further improving service quality and reliability. Despite these challenges in the development of new systems, the Agency's emphasis on interoperability and the introduction of components such as the European Search Portal (ESP) and Common Identity Repository (CIR) demonstrate efforts to enhance operational capabilities and secure data exchange among systems.

Overall, eu-LISA has demonstrated effectiveness in managing large-scale IT systems, with a focus on ensuring continuous service delivery, meeting SLAs, and addressing user satisfaction. However, ongoing efforts are needed to address identified challenges and enhance operational efficiency, particularly in the development of new systems and improving interoperability across platforms.

To what extent have the research and monitoring of research activities of eu-LISA contributed to the performance in delivering its core tasks?

Since 2018, eu-LISA has ensured regular publication of various research reports and other documentation that are accessible on eu-LISA website. 52 Between 2018 and 2023, the Agency has published 86 reports. These reports include technical reports, statistical documentation, and conference notes. Most thematic report series are published on an annual basis, others are produced irregularly.

The expansion of eu-LISA’s mandate widened the scope of the Agency from not only managing to also developing large-scale IT systems. The development aspect yields an increase in its role of monitoring and conducting research activities. In general, it is essential for eu-LISA to monitor and assess emerging innovative technologies to guarantee that the Agency can successfully achieve its strategic goals of maintaining secure and efficient external borders, and to implement state-of-the-art solutions within its systems.

However, research activities have arguably contributed to a very limited extent to the performance of delivering eu-LISA’s core tasks. This can be based on the following arguments. As mentioned above, most reports published disclose e.g. technical and statistical information and only 12% of the total reports published between 2018 and 2023 address thematic research topics, including feasibility studies. Hence, the number of reports contributing to the development of new systems is rather limited. Acknowledging the small number of staff dedicated to conducting research within eu-LISA, and considering the rapid pace of technological advancements and the emergence of new technologies, the overall resources -both in terms of staff and budget- are relatively limited. Given these constraints, eu-LISA’s resources dedicated to monitoring of research should be prioritised on areas directly related to the systems developed and managed by the Agency, thereby generating greater added value. By refocusing its research activities, the Agency could increase its focus on future needs and challenges such as Artificial Intelligence, biometrics, or quantum computation that had been stressed by survey respondents as potential future opportunities.

Has the Agency effectively performed its tasks related to developing new systems in line with its current mandate? How effective was the Agency to provide the technical capacity, to develop the systems? How effective was the governance structure provided by the Agency to support the development of the systems?

Besides the operational management of existing large-scale IT systems, eu-LISA is responsible for the development of systems if requested by the Union. According to the EU Regulation 2018/1726, the Agency is obliged to govern tasks relating to the development of new systems as laid out in specific Union legal acts as well as providing training on the technical system use (Chapter II, Article 9). This question concerns the effectivity of eu-LISA developing new large-scale IT systems in light of its governance bodies (i.e. Management Board, Programme Management Boards, and Advisory Groups).

Based on evidence, the development of systems is considered less effective. Even though sufficient technical capacity is available, recent developments suffered from significant delays. Moreover, the provision of support from the governance bodies, which became more complex over time, was effective only to a moderate extent.

The technical capacity to develop these systems has been acknowledged by stakeholders, with a majority indicating that the Agency provided the necessary expertise to a moderate to very large extent. However, the Agency's success in contributing to the timely development of new large-scale IT systems appears to be moderate to low (39 out of 80 respondents, 49%), as per the feedback from the Advisory Group members.

Even though the governance bodies of eu-LISA have been evaluated as internally coherent to a large extent, there are identified areas for improvement, particularly in speeding up decision-making processes, ensuring representation from the judicial domain, and enhancing communication and collaboration with the Advisory Group and Member States, including with eu-LISA’s external contractors. This highly resonates with several inputs received from interviewed stakeholders. While the governance structure was described as functioning well overall, there are some issues including, for instance, its complexity. Some interviewees stressed the rising number of different governance bodies in the past years and a resulting increase in workload for different stakeholders. In addition, the inclusion of justice systems into eu-LISA’s portfolio was stated to have contributed to this complexity, including because the bodies now represent different communities.

Has eu-LISA effectively handled incidents on each large-scale IT systems operated by the Agency?

eu-LISA's management of incidents across its large-scale IT systems reflects a robust framework with generally positive outcomes, although some challenges have been identified.

Survey responses indicate a high level of satisfaction with the handling of incidents, particularly highlighting fast response times and efficient incident management. However, concerns have been raised regarding system interruptions and data entry problems, highlighting areas for improvement.

Examining specific systems under eu-LISA's operation provides further insights. The SIS and VIS have demonstrated high availability rates, with minimal disruptions to operations For the SIS, in the year 2022, the system had a high availability rate of 99.94% accompanied by a response time of 98.98%.

For the VIS, it demonstrated a good level of availability, with the central system achieving an availability rate of 99.6%. Moreover, in terms of response time performance, the VIS central system showed a performance indicator of 99.7%. Overall, the VIS central system demonstrated a strong level of availability and responded efficiently to user requests.

No critical security incidents occurred during the reporting period. The Agency's security unit continues to maintain and develop security measures, operating its Information Security Management System (ISMS) in compliance with ISMS standards and ISO27001.

The VIS experienced a major upgrade on 20 May 2025 to ensure interoperability with the EES, migration to a Common Shared Infrastructure, and integration with sBMS. These efforts were complemented by a reassessment of security risks and the implementation of new security and business continuity plans.

Similarly, the Eurodac system has exhibited exceptional reliability and responsiveness, with an availability rate of 99.9% and efficient response times.

Interoperability readiness is a key focus for eu-LISA, with integration efforts underway to enhance data exchange and collaboration between systems 53 . Security measures remain robust, with proactive initiatives like the establishment of a Computer Security Incident Response Team (CSIRT) and cybersecurity exercises to strengthen incident response capabilities.

The Agency's support function, including 24/7 monitoring and effective incident resolution, has been lauded by the Member States (e.g. in the SWOT workshops and in several interviews). Additionally, both, end-users as well as Management Board and AGs express satisfaction with the support provided through the survey, underscoring the importance of eu-LISA's role in ensuring the smooth functioning of IT systems at both national and EU levels.

Overall, eu-LISA has showcased a very good track record in incident management, with a focus on timely response, high availability, and proactive security measures.

Does eu-LISA Regulation allow the Agency to cooperate with all international organisations, which are relevant for carrying out its tasks efficiently?

According to Article 43 of eu-LISA Regulation (2018/1726), the Agency is legally entitled to cooperate with international organisations that are relevant for eu-LISA to carry out its tasks efficiently if certain conditions are met. More specifically, according to Article 43.1, such cooperation must be provided for by a Union legal act. Moreover, the cooperation must be necessary for the performance of eu-LISA’s tasks. For all working arrangements established with international organisations or bodies, Article 43.2. underlines the need to specify terms of scope, nature, purpose, and extent. Furthermore, the working arrangements can only be concluded with the authorisation of the Management Board. Prior to such authorisation, the Management Board is required to receive an official approval by the European Commission.

For cooperations in the context of e-CODEX, technical support shall be allowed to be requested and received by the cooperation parties. This shall be handled by one natural person per organisation. 54

Has eu-LISA effectively implemented the recommendations of the previous external evaluation completed in 2016?

The 2017 evaluation of eu-LISA proposed 84 specific recommendations to eu-LISA, particularly on the Agency’s legal framework as well as on structural, organisational changes. 55 The recommendations focused on improving current tasks and addressing possible future needs of the Agency.

Overall, eu-LISA has incorporated the recommendations of the 2016 evaluation to a mixed extent. In addition, the recently introduced operational restructuring within the Agency aims at further implementing recommendations concerning staff matters and agility. Moreover, the management of eu-LISA’s Information Security Management System (ISMS) was strengthened and considerable resources have been invested to bolster information security across its spectrum of activities. The Agency carries out thorough security evaluation on the systems it manages. In addition, with the publication of technical and statistical system reports, eu-LISA addresses the 2016 recommendation of regularly reviewing systems’ capacity needs. For instance, since 2018 the Agency published in total 29 reports on technical aspects related to the different IT systems. However, there is still room for improvement in terms of eu-LISA’s research activities and access to Member States system data and audits to ensure adherence to prescribed standards and directives, thereby fortifying the ISMS.

Recommendations, which were considered as “critical” in the previous evaluation, which have not yet been fully addressed, include, for instance, the improvement of a backup site for switch over/switch back sequences between systems, a more detailed accounting system to better manage budgeting of resources or to strengthen the security management competencies.

Regarding the use of the backup site based in Austria, technical experts stressed during the interviews that due to the physical distance between Strasbourg and St. Johann im Pongau, an uninterrupted service in case of system failure cannot be guaranteed as switching to the backup site is taking too long. Moreover, the recommendation relating to accounting practices have not been addressed to a satisfying extent. The current processes have been criticised by interviewees for their lack of transparency. Another significant concern that remains to be addressed is the absence of a robust accounting and planning framework. This has been pointed out as a critical issue, making it challenging to project and manage finances effectively in the medium and long term.

Other recommendations classified as “very important” that have not yet been addressed include, among others, the definition of a sourcing strategy to improve procurement, to leverage the technical expertise in the Advisory Groups or to establish and formalise a review process allowing for a regular review of the systems’ capacity needs. Importantly, the Agency still lacks a common sourcing strategy. A noticeable gap has been identified between the human resources and procurement divisions located in Tallinn and the operations department in Strasbourg. This division has reportedly led to various inefficiencies over recent years up to the present, particularly in the processes of recruitment (e.g. length of procedures) and staff development (e.g. contract prolongation), as well as in overseeing contractors (such as selecting, managing, and monitoring contractor performance). Furthermore, as concerns the technical expertise within the Advisory Groups, stakeholders pointed to slow decision-making processes within the AGs, given that consensus must be reached. Moreover, once a decision is made and communicated to the Management Board, technical expertise is often missing in the latter governance body and an adequate problem assessment is challenging.

Hence, to conclude, while many of the recommendations have been addressed, key outstanding recommendations include:

-Address technical challenges stemming from the split between the Central Unit in Strasbourg and the backup site in St. Johann im Pongau to ensure uninterrupted services, hence, consider another back-up site in closer reach.

-Define a sourcing strategy. Based on such as strategy, the procurement strategy and principles for involving outside competences should be defined; and

-Establish and formalise a review process allowing for a regular review of the systems’ capacity needs with Member States based on the statistics generated from the systems and a prospective exercise.

4.1.2.Efficiency

The assessment of the Agency’s efficiency covers a) the financial resources and planning, b) the operational management of IT systems, c) the development of IT systems, and d) the governance and organisational set-up.

To what extent has the governance structure of eu-LISA supported its ability to perform its tasks, having regard to its size, composition, organisation and working processes?

As regards the efficiency of eu-LISA’s governance bodies (i.e. its MB, AGs, PMBs) and their structure and how this structure has supported the ability of eu-LISA as an organisation to perform its tasks, the evaluation found that the Agency has overall successfully managed to perform its tasks, considering the size, composition, organisation and working processes of eu-LISA’s governance structure.

Concerning the size and composition of the bodies, the evaluation found that Member States were able to contribute to the Advisory Groups and the Management Board. Participation by Member States varies which can lead to an underrepresentation of certain points of view. The representation of only some Member States in the PMBs was considered critical, and some Member States pointed to a lack of participation possibilities in terms of upcoming systems. The size of the Advisory Groups and the Management Board was considered by some stakeholders to negatively impact an efficient decision-making, given the need to align multiple different positions among sometimes more than 30 actors.

As concerns the organisation and working processes of the different governance bodies, the evaluation found the coordination between the bodies sufficient: while the Advisory Groups discuss the more technical questions, the Management Board is responsible for the overall decision-making and the Programme Management Boards are responsible for the preparation on new systems and acting as an intermediary between the other two bodies. In this context, the e-CODEX PMB is an exception as, unlike other PMBs, it is established as a permanent body, ensuring ongoing support, oversight and guidance for the e-CODEX system's development and long-term sustainability. This permanence is crucial in ensuring the integrity and effectiveness of the e-CODEX system, particularly in safeguarding the respect of the principle of the independence of the judiciary within its framework. This difference is accentuated by the absence of e-CODEX PMB members in the MB, which stems from the decision taken within the e-CODEX PMB aiming at safeguarding the principle of independence of the judiciary. Nevertheless, as a result the e-CODEX PMB Member are less familiar with the Agency's broader operations. The evaluation pointed possible improvements concerning unclear distribution of responsibilities, as well as the significantly high administrative burdens (e.g. for chairs) resulting from the coordination needs between the bodies.

How efficient is the Agency governance structure to reach its goals and complete its tasks?

The evaluation assessed the high-level implications of the governance structure of the Agency to reach its targets and complete its tasks. The evaluation found the governance structure to be overall efficient. The three primary governance entities – the Management Board, Programme Management Boards, and Advisory Groups – were found to be successful and functioning efficiently, with the implementation of decisions made by these bodies operating efficiently. Stakeholders highlighted that the representation of all Member States and EU Agencies using the systems in the Advisory Groups and Management Board to be a major asset, helping the end-beneficiaries of eu-LISA to raise a voice on how the Agency’s overall goals could be reached better.

However, the evaluation pointed to the challenges that the increase of governance bodies – in line with the increasing tasks of the Agency under the new mandate – raised, leading to an increasing complexity of the governance structure in eu-LISA’s overall set-up. In particular, the governance structure was seen as not sufficiently reflecting the interoperability of the systems, as the bodies as well as eu-LISA’s internal organisation continue to operate largely on a system-centric basis. These challenges concern:

-The complexity of the governance structure: the number of governance bodies has increased exponentially in the past years. The inclusion of justice systems into eu-LISA’s portfolio was stated to have contributed to this complexity, as the governance bodies now represent different communities. This led to a high administrative burden as the coordination between the different bodies can be extremely challenging and time-consuming. In general, the AGs and PMBs meet in different frequencies, varying between monthly meetings (e.g. ECRIS-TCN and EES-ETIAS AG) and four times per year (e.g. SIS AG). 56 For AGs meeting on a monthly basis, there is a constant need to prepare the meetings (agenda, sharing of materials), follow-up on meetings (executive summaries and detailed minutes) communicate, collaborate, and coordinate. Since the expansion of the Agency’s mandate to the development of new systems, the time and resources needed to deliver on this managerial task has significantly grown. Overall, there can be up to around 200 different meetings per year.

-The potential risks of duplications or contradictions: while eu-LISA’s Regulation allows for the cooperation between the AGs (Article 27(5)), in practice, the different bodies work separately and can lead to instances of inconsistent decisions among bodies, such as contradictory opinions in different AGs about the same matter, and the same topic may be discussed several times. While it is the role of the Chairs and of the Agency to note and react on inconsistencies, such inconsistencies cause additional work. Coherence in the opinions of the AGs is necessary to ensure interoperability between the different IT systems. Stakeholders also underlined that Programme Management Fora (PMFs) are being requested by Member States even in cases where there is no explicit legal base, highlighting the practical demand for mechanisms to coordinate governance and cross-system developments.

-The practical work of the different governance bodies: challenges relate to the expertise of the members of the MB and AGs across Member States and thematic fields, misunderstandings of the MB members of eu-LISA’s financial rules or the operational work of eu-LISA. This silo approach is reinforced by limited coordination between the different silos at Agency level, while at the same time national coordination mechanisms could also be strengthened to ensure more coherent participation and input across governance structures.

The increasing complexity also impacted the administrative burden and potential risks for duplications and/or contradictions in the Agency’s work. While the current governance structure may still be sufficient for eu-LISA to reach its goals at present, a potential change of the mandate on this point could reduce the complexity of the governance structure.

Consultations carried out with stakeholders suggested considering reducing the number of separate bodies by moving away from a systems-based approach to a clustered or sectorial approach. Suggestions referred to clustering the governance bodies by area of operation, namely (1) border management, including EES, ETIAS, VIS; (2) migration and asylum, including Eurodac, DubliNet; (3) law enforcement and customs, including SIS, API and PNR, and Prüm; and (4) justice, including ECRIS-TCN, e-CODEX and the JITs CP. In this context, several stakeholders representing their Member State in the governance structure of the Agency highlighted that they currently considered the justice area to be underrepresented within the governance bodies of the Agency, which might reduce the effectiveness and efficiency in the fulfilling eu-LISA’s tasks in the justice sector.

Other stakeholders highlighted that the governance structure should considered the individual systems’ specificities relating to their set-up and that the profiles responsible for the system at the national level may vary. At the same time, governance currently reflects a rather decentralised approach to the implementation of systems, with experts focusing on the priorities of their respective systems, which may not always align with a common overarching goal. The evaluation did not find evidence suggesting that the proposed set-up of clustering the governance bodies by area of operation would be more efficient. Therefore, looking forward, it may be worth exploring an alternative approach to the development of systems, potentially moving towards a more centralised governance model and considering a different involvement of national bodies to support more coherent decision-making and alignment across systems.

To what extent have eu-LISA’s organisational solutions, including outsourcing strategies, affect the performance of the Agency in developing the new systems?

The evaluation found that the organisational strategies adopted by eu-LISA, ranging from various solutions including outsourcing, have yielded a mixed impact on the Agency's ability to develop new systems.

Over the past years, the evaluation found that eu-LISA largely outsourced the work related to the development of new IT systems. This resulted in a series of challenges concerning the delivery of these IT systems in appropriate quality and efficiency, ranging from the complexity of the previous and current outsourcing approach, the division in roles and responsibilities between eu-LISA and external contractors, the contractual management, to the volume of outsourcing. Stakeholders note that eu-LISA does not involve contractors in governance meetings with Member States, limiting transparency and exposure to the wider governance bodies.

The Commission’s Internal Audit Service report of 2020 57 found that, compared to other Agencies, the proportion of structural service providers was high and, in addition, structural service providers were not only used for temporary or ad-hoc duties but also for core operational and/or administrative tasks. While there was a steep rise of eu-LISA staff since the adoption of the legal mandate in 2018, the number of tasks and the complexity related to the development of new large-scale IT systems has also increased, and the evaluation showed that the Agency still did not possess sufficient personnel to handle all these tasks within the Agency itself. In addition, throughout the evaluation period, eu-LISA was in strong competition with the private sector and other EU institutions as concerns the filling of vacancies for highly skilled IT staff, fuelling the problems to recruit enough staff to fill all open vacancies. Hence, though a controversial decision within eu-LISA itself, the Agency had to continuously pursue the outsourcing of core tasks to external contractors.

Furthermore, the work conducted by eu-LISA is regulated through the relevant legal instruments. The results-focussed perspective of some legal requirements in practice forced the Agency (i.e. not through the legal text itself but through the operational practices chosen by the Agency in response to the legal text) to an outsourcing model in which IT developments were contracted according to the waterfall principle instead in an agile fashion. This resulted in lengthy development cycles and difficulties adapting to changing needs or technological advancements, ultimately impacting project timelines and quality.

The evaluation showed the lack of a strategic approach for the outsourcing of resources. Due to the historic development of the Agency and its overall set-up, this has led to a situation in which outsourcing was, and still is, the default modus operandi of the Agency. This has led to challenges with regard to the development and delivery of new systems, such as EES, which was the Agency’s very first own development. However, in recent months, there has been a noticeable evolution, with eu-LISA taking steps to adapt its approach towards outsourcing, including greater integration and co-location with contractors to improve supervision and control.

SIS, VIS, and Eurodac were systems developed before the creation of the Agency, and the Agency has been entrusted with the responsibility to manage these three systems operationally. Therefore, originally, the Agency used a system-centric silo approach to outsource work related to SIS, VIS, and Eurodac. The same approach was taken in relation to EES, with the main purpose to externalise any substantial risks. However, stakeholders have pointed out that the Agency was not sufficiently equipped in recent years to manage, guide, and control the EES contractors and that the contractual vehicles themselves were not well-equipped for the development of an IT system. 58

The Agency changed its outsourcing approach from a system-centric silo approach with a limited number of consortium partners per system to a more horizontal approach of ‘layered’ outsourcing. Under this model, different types of services (related to e.g. operations, infrastructure, application, testing, development, maintenance etc.) can be provided by different consortia of contractors for each system. The restructuring indicates a shift from the limitations posed by isolated silos, such as fragmented service provision, lack of standardised operations, and reduced interoperability. The aim is to allow for more agile, standardised, timely, and cost-effective service provision.

However, the implementation of this approach has encountered some limitations. For example, under this approach, technical experts who had been hired based on their technical expertise, skills, and experience, have been assigned to contract management roles in which they were not able to apply their specialised skills. Challenges appeared in managing the transitions between different phases of outsourcing contracts (“sunset period”). The absence of dedicated internal transition teams accompanying contractors have contributed to complexities in the handover process (e.g. concerning decisions that still need to be taken, operational developments still outstanding, potential areas of improvement, testing results, bugs in the software, etc.), leading to disruptions and inefficiencies, risking delays in the development processes of large scale-IT systems. The establishment of internal transition teams emerges as a crucial aspect for enhancing eu-LISA's operational efficiency and project management.

Therefore, the evaluation found that the Agency did not strike an appropriate balance in recent years between delivering technical developments inhouse and simultaneously ensuring that contractors deliver on time in the necessary quality. The evaluation emphasises that the Agency should maintain strong internal control over system development, keeping key technical expertise and developments in-house, even while leveraging external contractors.

To improve the situation, the evaluation highlighted the importance for the Agency to increase the share of insourced work to reduce dependency on external contractors, and investing in internal capabilities – namely hiring (more) technical staff that is actually tasked with technical work for which their expertise is needed, rather than managing contracts – to enhance efficiency and oversight. A clearer strategic approach to outsourcing and improved contractual management, to mitigate risks and ensure the timely delivery of quality systems, are considered necessary by the evaluation given the escalating complexity of the IT systems entrusted to eu-LISA.

The evidence gathered for this evaluation showed certain positive developments on this topic. For instance, eu-LISA has transitioning towards a more integrated outsourcing approach and strengthening supervision over contractors and started to invest into co-sourcing with contractors (i.e. a closer, more integrated working relationship with contractors, for instance, on-site alongside eu-LISA staff) and implement the policy that everyone has to work ‘under one roof’. This means that contractors carry out developmental work related to the new system on premise in Strasbourg and provides eu-LISA staff with increased supervision and control possibilities. However, achieving the optimal balance between in-house development and outsourcing remains a challenge.

How efficient have been the management of resources by eu-LISA? To what extent has eu-LISA efficiently used its human and financial resources to ensure the development and the effective, secure and continuous operation of large-scale IT systems? To what extent has eu-LISA succeeded in building up in-house capabilities for handling various tasks entrusted to it?

The evaluation’s assessment of eu-LISA's management of resources, particularly its human and financial resources, revealed a mixed picture of efficiency and challenges.

The evaluation found that the Agency demonstrated success in certain areas, such as utilising advanced technology for central IT systems such as SIS and VIS, ensuring effective, secure and continuous operation, and fostering coordination among Member States and providers.

However, while the management of existing IT systems contributed to a coordinated and cost-effective environment, the evaluation pointed to moderate successes and challenges associated with the development of new IT systems and the handover/takeover processes. Challenges arising from slow bureaucratic processes and unclear tasks have been identified, indicating inefficiencies in project management.

Outsourcing practices, although moderately successful, were criticised for their complexity, contractual management, limited accountability and volume. The evaluation found that the Agency's default reliance on outsourcing, without a balanced approach to in-house development, resulted in difficulties in delivering tasks related to the development of new systems (see previous question).

Nonetheless, the evaluation also found that the Agency made progress in building its in-house capabilities to manage various tasks. Its recruitment efforts and HR processes have been digitalised, with initiatives like Agile Recruitment and video interviews. However, challenges remain, such as legal constraints on adopting new technologies and an inflexible organisational structure. At the same time, ongoing projects such as the cultural transformation and employer branding show eu-LISA's dedication to improving its capabilities and staying competitive in the IT field.

The evaluation found that the financial planning and reporting processes showed improvement over the evaluation period.

The distribution of resources and, accordingly, competences and responsibilities, is allowed between IT systems in relation to staff, as well as infrastructure and operating expenditures (administrative costs). However, the operational expenditures – by far the Agency’s largest budget item – are allocated according to systems. 59 This is considered to be a deterrent factor for the distribution of resources across systems and, thus, a challenge to practically respond to ad hoc issues and urgent needs. Particularly noteworthy within this context is the role of e-CODEX and the obligation to respect the principle of the independence of the judiciairy. 60 Such a full separation of resources can lead to silo-based operational approaches, and eventually cause inefficiencies as concerns knowledge transfers within the Agency.

The evaluation found that the increasing number of tasks of the Agency posed particular challenges concerning the planning. While initially eu-LISA was an Agency set up to manage three systems, it has become an Agency to manage and update three systems and, in parallel, develop additional new ones from scratch. While the appropriate planning capabilities were eventually established, this led to gaps between the actual and planned progress over time.

The evaluation pointed to the need for eu-LISA to professionalise its accounting and planning function through the implementation of appropriate tools. 61 In this regard, multi-annual planning is an area for improvement, both from a strategic perspective and the concrete tooling. Over the previous years, emphasis was given to the annual planning which led in the Agency to ask for too many financial resources, which could, ultimately, not be used entirely. This is reflected in the budgetary absorption rates (see Figure 4 below). 62 While particularly low absorption rates in the past stemmed from delays in the development of the EES (and sub-sequent delays in the payment of contractors), over the last years, the Agency was able to improve its budgetary absorption.

Figure 4: Evolution of budgetary absorption of eu-LISA, 2018-2024, in million EUR

Source: eu-LISA, Annual Accounts, 2018-2024.

Are the resources (human, financial and technical) used for the operational management and maintenance of existing systems adequate in comparison with the results achieved? Could the efficiency gains be obtained with a different approach to outsourcing or with different technology choices?

The evaluation found that the operational management and maintenance of IT systems remains one of the Agency’s strong areas. eu-LISA implemented these tasks through a combination of internal capacity and outsourced support (human and financial). Whereas the 24/7 management of incidents was done by internal staff, the operational support of users is outsourced to external service providers. This was, in principle, considered adequate by the evaluation: while challenging and security-critical tasks were implemented by staff, the more routine tasks have been outsourced.

As there only very few incidents occurred during the evaluation period, with existing systems running smoothly, the evaluation assessed whether or not staffing the 24/7 operational management of the systems with internal resources was the most efficient. While such activities are undoubtedly security-critical, they can also become routine, raising the issue of whether internal resources are always the best fit for these functions, particularly in light of challenges in recruiting for round-the-clock services and shortages of staff in system development. In this respect, it is key that the Agency carries out a clearer assessment of which tasks should be maintained intramuros and which ones can be more effectively externalised, so as to maximise the use of available resources.

The Final audit report on staff recruitment, management, and engagement of structural service providers and organisational ethics in eu-LISA, prepared by the European Commission Internal Audit Service in 2020 expressed similar concerns, noting that ‘the Agency needs to improve the methodology on which management and stakeholders reach decisions on the use of its staff versus the use of structural service providers in the production of its deliverables. The documentation of the justification of the current mix (staff versus structural service providers) is still weak. Compared to other agencies, EU-LISA has a relatively high proportion of structural service providers, and they are not only used for temporary or ad-hoc duties but also for core operational and/or administrative tasks’. 63 This remarks were also echoed in 2024 in the Final audit report on procurement and contract management in the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA), noting that, in relation to eu-LISA 2023 Consolidated Annual Activity Report, ‘weakness still exist in the sourcing of human resources, controls over procurement and contract management activities in the context of operational programmes and projects’. 64  

As concerns the choice of technical resources, the evaluation found that eu-LISA did yet establish a clear strategy and roadmap for the implementation and practical use of artificial intelligence within the operational management of systems (e.g. concerning incident detection and management, as well as the supportive communication with users through chatbots). Tasks were performed manually, blocking financial and human resources that could, eventually, be better used elsewhere. The evaluation therefore found that the technology choices of eu-LISA were considered to be an important lever to increase the efficiency of the Agency in this regard.

Are the resources (human, financial and technical) used for the development of new systems adequate in comparison with the results achieved? Could the efficiency gains be obtained with a different approach to outsourcing or with different technology choices?

In view of the delays and quality-related issues in relation of the delivery of the new systems, the evaluation considered that the resources used for the development of new systems were only partially adequate. Efficiency gains could have been acquired through a larger share of insourcing development work and more proactive investment into new technologies.

The development of new systems was largely outsourced over the past years. This has led to severe delays and quality concerns, in particular with regard to the delivery of EES. A significant portion of the Agency’s financial resources were spent on external contractors that were not fully able to deliver, while at the same time too few resources were spent of staff guiding contractors or co-developing systems.

In addition, technical eu-LISA staff was mainly in charge of project management tasks contractors rather than performing technical work: A job for which specific expertise and experience, as well as the right internal processes and contractual levers are needed in order to be effective. As a consequence, eu-LISA was not fully able to deliver on the task to develop new systems in alignment with the expectations of stakeholders.

Therefore, eu-LISA could have been more efficient with regard to the development of systems if it had chosen a different approach to outsourcing: Rather than silo-based outsourcing or even using the layered approach under TEF/TOF (Transversal Engineering Framework/Transversal Operation Framework), better results could eventually have been achieved through increased insourcing of resources and better leveraging the technical expertise already available within eu-LISA.

For this purpose, the evaluation identified two crucial factors for future improvement: The introduction of a portfolio management approach within eu-LISA, as well as the definition and execution of a long-term sourcing strategy (‘make and/or buy’).

To what extent are procurement, finances, personnel and any other corporate services organised efficiently? Are resources commensurate with results achieved?

The evaluation of the efficiency and organisation of procurement, finances, personnel, and other corporate services within eu-LISA showed a blend of strengths and areas for enhancement. The Agency's heavy reliance on external resources facilitated the management of workload demands associated with both legacy and planned systems. However, whether resources were commensurate with the results achieved presents a mixed picture.

The evaluation highlighted the importance of enhancing compliance and strengthening procurement processes. Main issues identified included the need for additional procurement managers to handle workload demands effectively, for more detailed attention to the technical requirements outlined in the terms of references to ensure successful procurement outcomes, and for clearer processes to navigate the complexity of contractual management. Recommendations from various stakeholders included exploring creative procurement approaches, such as outcome-based contracts, to align with the fast-paced nature of the technology market.

However, despite ongoing efforts, the evaluation pointed to continuous challenges faced by eu-LISA in its procurement processes. Inconsistent stakeholder responsiveness and a lack of clearly defined processes were identified as challenging factors. For instance, variations in stakeholder engagement and input can lead to delays and discrepancies in procurement decisions. Without well-defined processes outlining roles, responsibilities, and timelines, a risk of inefficiencies and misunderstandings in the procurement workflow appears. The shift towards alternative procurement models, such as agile delivery, was suggested to address these challenges and increase control over service delivery. 65

The evaluation pointed to the need for additional human resources in certain departments, particularly in activities related to the Operations and Corporate Services Departments. 66 Reporting discrepancies, extreme dependency on contractors, and challenges in recruitment and retention were found by the evaluation to contribute to resource shortages. These challenges can be effectively addressed through a more strategic and integrated use of existing resources.

The evaluation also found that delays and challenges in project delivery are an indication of potential inefficiencies in resource allocation and management. Furthermore, complexities within the organisational structure and governance processes highlight the need for streamlining operations to enhance resource efficiency. To address these disparities effectively, resource distribution strategies could be reassessed to ensure optimal outcomes. Overall, the evaluation finds out that that eu-LISA should optimise internal resource allocation, avoid over-reliance on external partners, and maintain integrated oversight of project development rather than leaving different aspects to external service providers.

In this context, challenges related to the organisational split between Tallinn and Strasbourg were also noted in the evaluation as a factor impacting collaboration, communication, and operational efficiency. The perceived disconnection between corporate services and operations highlights the need for improved clarity of roles and responsibilities and closer alignment between units.

Additionally, several interviewees expressed concerns about staffing levels and contractual arrangements, which underscores the importance of addressing recruitment and retention challenges, while adapting to evolving work preferences, including remote and flexible working arrangements. Stakeholders noted that the Agency tends to keep the individual allocation of resources and personnel tied to systems rather than managing projects in a more integrated and coordinated manner.

Overall, while eu-LISA demonstrated transparency through its published information and efforts to strengthen procurement and compliance, whether these resources were commensurate with results achieved present a mixed picture. Strategic resource management should emphasise optimal internal use, coordination across systems, and retaining control over critical project aspects to ensure efficiency and effectiveness. Additional, addressing concerns about claimed resource shortages, improving organisational effectiveness, and enhancing collaboration between units are critical priorities for optimising efficiency and achieving desired outcomes. This is particularly important for units such as the Corporate Services Unit and Program and Project Delivery Unit, ensuring that allocated resources align effectively with the Agency’s objectives..

To what extent has the tripartite set-up of eu-LISA affected the efficiency of the use of resources by the Agency? How does eu-LISA's split location affect its activities? How has eu-LISA ensured efficient working on three separate locations?

The evaluation did not find the split-up of the Agency to be a major hindrance in terms of resource efficiency. The evaluation noted that the split-up can lead to additional organisational costs, given the need for frequent travelling and the establishment of additional coordination activities, and increased costs for maintenance and office rents.

The evaluation found that, overall, eu-LISA has found a way to efficiently handle the split-up of its locations. The separation of tasks reduced the need for administrative coordination, allowing for the processing of tasks largely within a certain location. Teams are rarely split-up, ensuring connectivity of the team members.

In summary, while the evaluation found the Agency’s efficiency is not significantly impacted by the split over different locations, improvements in terms of additional teaming activities and better online coordination tools could be implemented to mitigate for certain effects of the tripartite location of the Agency.

Having regards to the organigram of the Agency, the distribution of competences, and the sharing of responsibilities between the different entities, to what extent does the internal organisation of the Agency impact the efficiency of eu-LISA to reach its objectives?

The evaluation found that eu-LISA’s internal organisation as satisfactory, with a high approval rate by stakeholders of the efficiency of the internal structures, escalation procedures and decision-making procedures. The evaluation pointed at two aspects of the Agency’s internal organisation which could be improved: (i) the distribution of responsibilities and competencies and (ii) the teaming and coordination between the two locations in Tallinn and Strasbourg.

The evaluation pointed to unclear roles and overlapping responsibilities and competencies among different units and sectors within the Agency, leading to inefficiencies and coordination issues. Despite having bottom-up descriptions, there was a lack of clear top-down strategic definitions of roles and responsibilities in several departments. For example, three separate units have governance bodies within their responsibilities, with some overlap between the executive support and stakeholder relations unit and the liaison office in Brussels. In addition, multiple entities are responsible for programme and project management, creating potential for overlap and the need to clarify these roles. Some of these overlaps and role ambiguities have been addressed and clarified over the course of the evaluation period, contributing to improved coordination and operational clarity.

The evaluation also pointed to the loss of information that naturally occurs in coordination activities between the two main sites in Strasbourg and Tallinn. This concerned a perceived disconnect between operations (France) and both the HR and procurement units (Estonia). This has led to some inefficiencies in hiring procedures (e.g. length of procedures), development of staff (e.g. contract prolongation), and management of contractors (e.g. responsibility to select, manage, and control contractors). Moreover, the evaluation also noted that Strasbourg currently hosts more junior staff whereas the more senior staff is located in Tallinn, leading to an unequal distribution of knowledge within the Agency. However, despite certain hindrances, the overall cooperation between the two sites in terms of internal organisation is found overall to be satisfactory.

To what extent were the annual budgets of the Agency implemented in an efficient way and in view on achieving results?

The Agency’s budget is structured in three larger clusters: (1) staff expenditure; (2) infrastructure and operating expenditures; and (3) operational expenditure, as illustrated in Figure 5. Between 2018 and 2024, eu-LISA’s budget increased twofold, from EUR 152 million to EUR 307 million (compound annual growth rate of +13.8% annually) 67 . For the financial year 2024, staff expenditure accounted for 15% of the Agency’s budget, infrastructure and operating expenditure accounting for 8%, and operational expenditure accounting for 77%. This distribution indicates a strong emphasis on operational activities, reflecting the Agency’s priority to enhance IT systems and support functions while maintaining relatively lower administrative and infrastructure costs.

Figure 5: Evolution of eu-LISA’s budget from 2018 to 2024, in million EUR

Source: eu-LISA, Annual Accounts, 2018-2024.

The analysis of specific budget items from 2018 to 2024 (illustrated in Figure 6) indicates that expenditures on:

-IT system development increased by 661%, reaching EUR 76.1 million in 2024. This substantial increase aligns with the Agency’s expanded mandate to develop IT systems. However, this increase should be read together with the delays and quality issues in output (explained in the effectiveness section), indicating inefficiencies in this area;  

-Operational management increased by 362%, reflecting an EUR 167 million increase by 2024 compared to 2018. Despite this relative increase, eu-LISA demonstrated successful contributions in operational management, suggesting more efficient resource-use in this field.

-System security and business continuity, part of the operational management ledger, increased by 17%, while shared system infrastructure expenditure decreased by 297%, totalling EUR 3.39 million in 2024.

As regards staff (Figure 7), staff expenditure increased by 160%, an increase consistent with the increase in staff numbers. In 2024, eu-LISA had 404 staff, translating to an average cost of EUR 104 207 per staff member.

Figure 6: Evolution of eu-LISA’s expenditure on specific items from 2018 to 2024, in million EUR

Source: eu-LISA, Annual Accounts, 2018-2024

Figure 7: Evolution of eu-LISA’s staff and staff expenditure, 2018 to 2024, in million EUR

Source: eu-LISA, Annual Accounts, 2018-2024

Despite high budget implementation rates (100% for commitment appropriations and 89.7% for payment appropriations in 2024), the efficiency of budget allocation remains questionable due to the disparity between expenditures and results, particularly in new system development. The increased operational budget was better utilised, reflecting successful contributions, and administrative and infrastructure expenditures remained relatively low compared to other EU agencies, suggesting efficient resource management in these areas. However, bureaucratic barriers delayed budget allocation, leading to rapid and inefficient spending towards the end of budgetary years, indicating a need for more flexible and timely budgeting processes.

The evaluation also pointed out to the time-consuming nature of the annual budget allocation process, which often led to delayed and rushed spending. More importantly, the evaluation found that the Agency lacks comprehensive multi-annual planning capabilities, hampering efficient budget utilisation, with stakeholders noting the need for more strategic and less compliance-centric budget management. The iterative nature of IT development requires flexible resource availability, often constrained by rigid budget structures. Legal and structural constraints, such as the e-CODEX Regulation requiring full separation of resources, limit the Agency’s agility and contribute to inefficiencies.

In conclusion, while eu-LISA demonstrated adherence to EU budget rules and displayed financial diligence, there a is need to enhance its financial planning processes to ensure efficient implementation of annual budgets aligned with achieving results. The Agency would need to develop a comprehensive multi-annual overview and planning capability across its entire portfolio of tasks and services to establish a balanced scorecard, portfolio and pipeline management. A shift towards more strategic and flexible planning will better utilise resources, support overcoming bureaucratic barriers, enhance budget efficiency and maximising the Agency's impact.

Having regards to the Human Resource strategy, are the allocation of positions especially managerial as well as secretarial positions and the distribution of projects within the Agency’s staff in line with the Agency’s objectives?

The evaluation found there was no HR strategy in place by the Agency. eu-LISA used the Single Programming Documents as well as the overall eu-LISA strategy as guidelines on this matter. The evaluation pointed to the lack of capacities and resources allocation for the establishment of such strategy.

Figure 8: Evolution of posts and vacancy rate (2017-2024)

Source: eu-LISA, Annual Accounts, 2018-2024

The substantial expansion of the Agency’s mandate was accompanied by a sharp increase in the posts available (+46% from 2018 to 2019) 68 . In a similar fashion, the number of posts has increased steadily since then and more than doubled compared to 2018. At the same time, however, from 2019 to 2023, eu-LISA faced challenges concerning the recruitment, retention, and development of human resources, which has led a continuously high number and proportion of vacancies (reaching a high of 18,2% in 2022).

The distribution of tasks within the Agency referred to the overall tasks that needed to be performed in order to meet the overarching objectives. This included, among others, the management of the large-scale IT systems, the development of new systems, or conducting relevant research according to the scope of the mandate. Hence, a clear HR strategy would better align needs and capacities of and within the Agency, decrease individual workload, and improve the hiring of suitable staff.

In the past, the Agency adopted a delivery model where staff was primarily responsible for overseeing and ensuring the delivery of contractors' work (see also section above concerning outsourcing). Concerns have been voiced about the potential dilution of responsibilities, the creation of silos, the lack of accountability and a diminishing focus on in-house development and subsequently the knowledge-base. Hence, according to stakeholders from within the Agency, knowledge management became a challenge, with eu-LISA staff being overloaded with contract management responsibilities and in addition to risking the loss of institutional knowledge.

The evaluation found that Agency performed very well with regard to the operational management of existing IT systems, yet it struggled concerning the development of new systems, including due to an inefficient use and management of contractors.

The evaluation found that the Agency did not approach the management of human resources from a sufficiently strategic perspective. During the previous growth phase of the Agency, staff was hired to fill an ‘ever-expanding’ number of vacant positions. The challenges related to the development of new IT systems showed that eu-LISA did not yet possess clear strategic visibility about its human resources’ pipeline and needs.

Hence a holistic HR strategy, as well as the appropriate tooling with which the Agency could effectively plan, prioritise, and steer the allocation of its staff holistically according to functions, roles and profiles in view of existing and future business needs, rather than positions, is missing.

The evaluation also found potential for improvement with regard to the Agency’s internal knowledge management. To enhance knowledge management within the Agency, implementing a job rotation programme could prove beneficial. Such initiative would entail periodically rotating staff across diverse roles, systems, or domains, addressing the issue of knowledge silos and fostering knowledge transfer. By exposing staff to various facets of the Agency's operations, this approach would cultivate a broader perspective and would equip staff to address technical challenges from multiple angles. Furthermore, job rotation would present new challenges and avenues for skill development.

Lastly, a critical step in developing the Agency’s knowledge management would be a review and assessment of existing knowledge management processes, thus implementing a unified Knowledge Management strategy. Such an approach would streamline efforts, mitigate duplication of work, and ensure effective capture, sharing, and utilisation of valuable knowledge throughout the organisation.

Is the Agency’s Human Resources strategy adequate to attract and retain high-skilled profiles in the context of the IT labour market?

The evaluation pointed to a lack of a dedicated HR strategy document or roadmap and assessed the existing challenges with regard to IT talent attraction and retention stemming from this gap. The evaluation identified various challenges in relation to the recruitment and staff retention:

·Strategic factors: Availability of positions; Competition with private industry; Purpose and mission of the Agency (i.e. its branding); Location and facilities of the Agency;

·Contractual factors: Necessity of security clearance; the need of short-term contracts; Salary and compensation package. These issues, however, could be mitigated through strengthened human resource management, including more strategic planning and resource allocation practices;

·Operational considerations: Ability to work with cutting-edge technology; Rigid ways of working.

In addition, many stakeholders from within the Agency pointed towards process-related, contractual, and operational challenges, for example the bureaucratic nature and undue length of the hiring processes, types of contracts available to applicants, and the Agency’s limited ability to offer working with cutting-edge technology stack, in particular AI.

While the evaluation found that the Agency performed well with regard to the operational management of IT systems, it faced challenges concerning the development of new systems (see previous question), which could eventually have been avoided with more strategic workforce planning. The Agency’s adoption of a delivery model where staff was primarily responsible for overseeing and ensuring the delivery of contractors' work led to knowledge management challenges and had negatively impacted staff retention.

The evaluation found that eu-LISA did not approach the management of human resources, including of highly-skilled profiles, from a strategic perspective. A clear strategic visibility about the human resource pipeline was missing, as the recruitment was still being performed rather ‘position by position’ in order to deliver as effectively and efficiently as possible on its tasks while simultaneously avoiding shortages.

To what extent does the internal organisation of eu-LISA allow and provide the distribution of resources, competences and sharing of responsibilities between IT systems?

The evaluation found that the internal organisation of eu-LISA should allow for the distribution of resources, competences and sharing of responsibilities between IT systems. In practice, however, the existing structures within the Agency have led to team allocations specifically tailored for large implementation initiatives. While this approach enabled deep specialisation and holds the promise of optimal IT delivery and improved time to market, it also created knowledge silos and posed risks in terms of sharing expertise. This is particularly relevant given that one of the Agency’s core tasks involved ensuring interoperability across large-scale IT systems, yet its internal structure continue to reflect a silo-based approach. Additionally, heavy reliance on outsourcing further aggravated the issue, as it increased the risk of lack of knowledge-sharing and created dependence on external vendors. At this moment, the Agency depends on essential Subject Matter Experts (SME) and various documents to facilitate the sharing of expertise among the teams.

Enhancing eu-LISA's flexibility implies a conscious shift towards a more interconnected, agile and fluid organisational structure. It is important to mention that the Agency is already adopting agile practices, diminishing its dependence on traditional waterfall project management, which will further enhance teamwork and the exchange of expertise across large-scale programmes. As a consequence, eu-LISA has already introduced and is currently scaling the use of more agile squads and waves in order to break up existing silos.

Although operational work in relation to existing systems and systems under development is highly specific to each of the systems, there are also horizontal aspects that need to be aligned across systems. On the one hand, eu-LISA ensures the proper implementation of such aspects through allowing functional / horizontal procurement as part of TEF / TOF (Transversal Engineering Framework/Transversal Operation Framework). On the other hand, there seems to be only a practical separation between existing systems and systems under management. Eu-LISA staff only being able to work on one or the other makes sense, given that different skills, expertise and experience are required in relation to both of them.

However, with e-CODEX, there is also an exception: The e-CODEX Regulation specifies that resources devoted to e-CODEX may only conduct work in relation to this system and not to others. This is augmented through the e-CODEX Regulation specifying that the operational management of e-CODEX shall take place in Tallinn (which is not consistent with eu-LISA’s organisational set-up, as the IT facilities and relevant staff are based in Strasbourg). Hence, does not allow eu-LISA to apply its tested and proven business model but necessitates changes, both in the organisational structure, processes (including recruitment and budgeting).

Given that the human and financial resources allocated to e-CODEX remain comparatively small, these organisational restrictions could create limitations for maximising the system’s potential and, more broadly, for enabling the Agency to fully deliver on its objectives of increased effectiveness and efficiency.

Is the technology used by the Agency the most suitable to deliver its tasks? Is the Agency using the most advanced technology?

The evaluation of whether eu-LISA's technology is the most suitable for its tasks and if the Agency is using the most advanced technology paints a nuanced picture. While the evaluation noted eu-LISA's proactive approach in adopting cutting-edge technologies such as cloud, AI, and DevSecOps, operational constraints and regulatory barriers posed challenges to fully leveraging these innovations.

In particular, the Agency’s mandate and the establishing Regulations of IT systems include legal barriers to working with cutting-edge technologies such as cloud computing, quantum computing, and AI. For example, Article 5 of the eu-LISA Regulation provides requirements to process all data on-premise for all operations, which limits the flexibility in technology choices and introduces challenges common to large-scale IT systems leveraging on-premise solutions.

Despite these challenges, there is recognition of eu-LISA's pursuit of innovation and operational efficiency by key actors of the Agency, who see untapped potential of innovation that they are currently exploring. For example, the Agency is exploring AI-driven analytics for real-time monitoring and the integration of AI-based applications such as chatbots and LLMs (Large Language Models) for first-level support.

Survey responses further provide insights into various aspects of eu-LISA's technological landscape. According to the majority of responses (82%) the Agency has succeeded or largely succeeded inusing the best available technology for core systems SIS and VIS. 69  Additionally, there is a recognition of the importance of keeping up with new technologies, increasing research and pilot testing capabilities, implementing stronger feedback and cooperation mechanisms, and addressing new needs such as AI for identification and biometric data, cloud services, cybersecurity, agile workflow, communication, and test environment improvements (e.g. test automation).

To bridge the gap between aspirations and operational realities, the stakeholder consultations for the evaluation period reveals that the Agency should invest in strategic areas such as infrastructure maintenance and discuss a change of the mandate with DG HOME regarding the usage of sovereign cloud solutions. By enhancing infrastructure maintenance planning, investing in cloud technologies, and cultivating cloud-native teams, the Agency may unlock the full potential of advanced technologies to enhance operational efficiency and effectiveness, which remained an untapped potential in the past.

How does the tripartite set up of the Agency affect the allocation of Human Resources?

The evaluation assessed in particular how human resources of the Agency were allocated within eu-LISA and how the tasks differ between the different locations.

The evaluation pointed to a general understanding for and acceptance of the decision to locate the Agency’s sites in three Member States (with the liaison office being established in Belgium, a fourth Member State). Staff distribution across sites as of 31 December 2022 differs significantly, with the largest share of staff working in Strasbourg (73%), followed by Tallinn at 26%, while Brussels had a much lower share (1%) 70 . The tripartite set-up of the Agency mirrors the distribution of competencies within the Agency, with the site in Tallinn largely working on horizontal and administrative tasks, whereas the locations in Strasbourg hosting the technical staff and resources (e.g. central systems). The back-up site in St. Johann im Pongau is usually staffed on a rotating basis, with staff officially being located at different locations. The rest of staff is located in Brussels (of which most in the liaison office), working on publicity and lobbying-related tasks and ensuring the visibility of the Agency at the relevant institutions. 71  

While the reasoning of the split and the staff allocation was not challenged, the evaluation noted certain practical challenges, as the split of human resources across the locations could be problematic for example in terms of team spirit and understanding of the tasks of the staff in the other location. The most frequently mentioned challenges concerning Tallinn-Strasbourg were: (1) Detachment of corporate services from operations; (2) Remoteness and practical reachability of both Tallinn and Strasbourg; and (3) Secluded location of the Central Unit and the rented office space in Strasbourg. The evaluation found that the location of the Agency across separate sites mainly have had an impact on the hiring (e.g. length of procedures) and development of staff (e.g. contract prolongation), as well as the management of contractors.

The location of corporate services in Tallinn were perceived to lead to a lack of understanding among staff based in Tallinn for the actual tasks delivered in Strasbourg, as well as the needs and practical challenges of the colleagues there. In addition, the location of control functions in Tallinn (e.g. audit) posed certain practical challenges as there is only limited direct contact to those persons, processes, and data that should be controlled. While the collaboration between corporate services and operations was generally considered to work well, including through frequent (virtual) exchanges and the co-location of some units and sectors at both sites, the physical distance between Strasbourg and Tallinn was regarded as a cause of inefficiencies concerning the operational and financial planning, demand management (e.g. of positions, technical infrastructure and equipment), recruitment of staff, technical implementation of the Agency’s tasks (i.e. management and development of systems), and management of the contractors.

As the Agency is located in Strasbourg and Tallinn, with its back-up site in Sankt Johann im Pongau, similar concerns were voiced with regard to the collaboration and communication with the Commission based in Brussels. Specifically, a limited number of stakeholders from outside the Agency have highlighted the challenges stemming from the liaison office's role in Brussels, which, while originally intended to bridge policy-making in Brussels with the Agency's development in Strasbourg/Tallinn, has evolved into a more active and strategic entity (although the Agency does not have a mandate for policy-making). This expansion of eu-LISA’s activities, alongside overlaps with other units such as the executive support and stakeholder relations unit, has led to instances of inconsistent communication and increased coordination efforts, impacting operational efficiency.

As concerns the split of the two technical sites – the Central Unit in Strasbourg and the back-up site in St. Johann in Pongau, the distance between both sites brought about technical challenges with regard to the provision of uninterrupted services, particular concerning the availability, connectivity, latency, maintenance, and regular back-ups of the systems.

The evaluation found that the geographical location of the Agency’s sites posed challenges in terms of attracting and retaining staff. Tallin, in particular, was frequently highlighted by stakeholders as presenting difficulties for recruitment, while in Strasbourg, the Central Unit’s placement on the outskirts of the city was not seen as an asset for attracting talent.

As a mitigation action, the evaluation pointed to the possibility for eu-LISA to consider drafting a fully-fledged Human Resources and Procurement Strategy for the future, where the above-mentioned aspects would find consideration.

4.1.3.Coherence 

This evaluation criterion examined the extent to which eu-LISA’s activities were aligned in an effective and efficient way with those of other stakeholders, including the Member States and the Schengen Associated Countries, the European Commission and other EU institutions and agencies, to ensure coherence and avoid duplication of efforts. The evaluation also assessed the coherence of eu-LISA’s cooperation activities, management structure and activities, whether the Agency’s services were coherent with user’s expectations and needs, and the coherence of the Agency’s mandate with the other legal acts of the information systems it is responsible for.

To what extent is eu-LISA acting in cooperation with the European Commission and other EU institutions and agencies, as well as with Member States and Schengen Associated Countries to ensure coherence and avoid duplication of efforts?

Ensuring coherence and avoiding duplication is crucial to maintain the effectiveness and efficiency of eu-LISA, both in relation to the management of operational systems, as well as the development of new systems.

As regards the systems already implemented, the evaluation found that eu-LISA’s management of these systems contributes to a coordinated and coherent IT environment, with limited operational cooperation between the Agency and the Commission being necessary. However, as assessed in the efficiency section, the evaluation found challenges with regard to the strategic and budgetary planning and supervision of the Agency by the Commission. Users of the existing systems (Member States and SACs, other EU agencies) have expressed a high level of satisfaction with the Agency’s support and performance. An area for improvement identified by stakeholders is the governance of the agency, notably relating to the administrative burden the governance structure entails, the operation, cooperation and necessary expertise needed within AGs and PMBs (see section on efficiency on governance). The overall coherence of the governance structures of the Agency could be improved to further enhance the efficient management and operation of the Agency.

As regards the development of new systems, the evaluation found that the Agency has showed a constructive collaboration with Member States by being responsive to the feedback and suggestions for improvement of the new systems. The evaluation pointed to the importance of strategic and operational cooperation with the Member States, SACs, the Commission and other EU agencies to avoid gaps, duplications or grey areas in the future.

However, the evaluation found that certain ambiguities persist regarding the division of responsibilities between eu-LISA and other EU institutions, agencies, Member States, and private sector actors (e.g. contractors). In particular, the respective roles of the Agency vis-à-vis the Commission, Frontex, and Member States are not always clearly delineated. These uncertainties are also reflected in the budgetary dimension, notably in relation to the Legislative Financial Statements (LFS). While the LFS are negotiated on a year-by-year basis, stakeholders underlined the importance of strengthening the Agency’s internal planning for the use of the LFS, ensuring that resources are aligned with long-term strategic needs. The evaluation noted that eu-LISA currently lacks a dedicated tool or system to effectively support multi-annual planning and resource allocation, which limits the Agency’s ability to anticipate and coordinate needs over several years. Reliance on traditional annual budgetary cycles further constrains the effectiveness of coordination and communication, particularly during the preparatory stages of the LFS. Introducing more structured multi-annual planning mechanisms would help enhance clarity, predictability, and efficiency of the Agency’s resource planning.

To what extent are eu-LISA’s management structure and activities internally coherent for an efficient management and operation of the Agency?

The management structure covers both the Agency’s governance structure, together with its internal organisational set up (i.e. departments, units and sectors). eu-LISA’s management structure consists of the Management Board (and various sub-bodies), nine Advisory Groups 72 , as well as five Project Management Boards. In addition, the Management Board secretariat plays a central role in supporting the governance bodies and ensuring their smooth functioning.

The evaluation found that the management structure is internally coherent and contributes to the Agency’s overall effectiveness and efficiency. In particular, the involvement of all stakeholders in multiple governance bodies was underlined as an important success factor by Member States, as the responsibility on the on the continuous oversight and improvement of the respective IT system should lie with the “community of Member States”. However, the numerous parallel governance bodies in which all Member States and EU agencies who are users of the systems are represented, brings its own complexities (e.g. high administrative burden), risks of duplications and contradictions, and may lead to shortcomings relating to the efficiency of the governance structures, as outlined in the efficiency section.

The organisational set-up of the Agency evolved during the evaluation period. The current organigramme with its three departments, as well as sub-sequent 11 units and 35 sectors and three executive units directly reporting to the Executive Director was developed based on the Commission’s model in mind. While the organigram does not seem to have any obvious structural gaps, the evaluation found there is room for improvement with regard to the Agency’s organisational effectiveness and efficiency. For example, past situations were noted in which the practical roles and responsibilities of units and sectors lacked clarity, and instances occurred where multiple units worked on similar tasks without sufficient coordination. During the evaluation period, the organigram was revised, and several of the shortcomings previously identified have been addressed to some extent; however, some inconsistencies and overlaps remain, indicating the need for continued attention to organisational alignment.

In addition, stakeholders from within the Agency pointed out that the organisational split of the Agency between Tallinn and Strasbourg into corporate services and operations is bringing about challenges (e.g. a perceived disconnect between both the HR and procurement units in Tallinn and the operations department in Strasbourg) which have led to inefficiencies concerning the hiring and development of staff, as well as the management of contractors.

The evaluation found that, overall, the Agency’s management structure and activities are internally coherent. At the same time, opportunities exist to further enhance the management and operational effectiveness of the Agency, which is already actively pursuing a transformation aimed at increasing efficiency.

To what extent have eu-LISA’s services been coherent with the expectations of the users of large-scale IT-systems? How does the Agency’s service correspond to the needs and expectations of the Member States as users of the systems?

Services provided by eu-LISA include, among others, the development and operational management of large-scare IT systems in the area of freedom, security, and justice, creating an effective and coherent IT environment, or ensuring system continuity and security. The evaluation found that eu-LISA has met, to a large extent, users’ needs, including strong system availability, incident responsiveness, or operational support, with the provided large-scale IT systems within the scope of eu-LISA’s Regulation.

The evaluation identified areas requiring further attention by the Agency, such as improved search capabilities and interactivity in DubliNet, and addressing issues related to false positives, system scalability, interoperability, and search functionalities in Eurodac, VIS, and SIS. The need for enhanced pilot testing, stronger feedback mechanisms, and ensuring adequate resources and system security to meet future demands were other areas highlighted by stakeholders. Some stakeholders have expressed concerns regarding system usability improvements and data quality.

To what extent are procedures in place to ensure the coherence of eu-LISA’s cooperation activities with the policies and activities of its stakeholders?

The strategic direction of eu-LISA is encapsulated in its 2021-2027 agenda 73 , emphasising the importance of coherent interactions with stakeholders to meet the operational needs of essential systems such as SIS II, VIS, and Eurodac, while also undertaking the development of new systems such as EES, ETIAS, and ECRIS-TCN. These tasks are projected to shape eu-LISA's role as a holistic IT service provider across the entire JHA landscape.

The evaluation found that the procedures ensuring the coherence of eu-LISA’s cooperation activities with stakeholder policies and activities demonstrate a structured approach aimed at aligning with stakeholder expectations and operational needs.

The evaluation showed that the Agency employed a multifaceted approach to ensure alignment with both stakeholder policies and activities. This entailed engaging in day-to-day operational activities and actively participating in policy dialogues and strategic planning exercises with key stakeholders. By embedding itself within the policymaking process, eu-LISA can better anticipate and adapt to evolving policy frameworks, ensuring that its cooperation activities remain coherent with stakeholder policies. An enhanced coordination especially with the European Commission, is essential to ensure eu-LISA's activities are firmly aligned with its mandate 74 . Additionally, the Agency implements robust feedback mechanisms to gather insights from stakeholders, enabling it to fine-tune its activities in line with their expectations and requirements through the governance bodies. This proactive approach not only enhanced coherence but also fosters a collaborative environment conducive to achieving shared objectives.

In addition, the evaluation found that the Agency’s ongoing efforts to enhance stakeholder management strategies and improve the efficiency of service delivery, align with the Agency's strategic objectives and solidifying the coherence of eu-LISA's cooperations 75 .

To what extent are the activities of the Agency as well as the scope of its mandate coherent with the legal act governing the Information Systems the Agency is responsible for?

Albeit not impacting negatively on the coherence, the evaluation found that the complexity of the legal framework in which eu-LISA operates presents a significant challenge. This framework comprises the various Regulations governing each of the large-scale IT systems managed by the Agency and the eu-LISA Regulation itself. Interpreting these Regulations, and successfully developing, implementing, and operating the systems within the accompanying regulatory, budgetary, and organisational constraints, has proven to be a demanding task.

The evaluation has not found major incoherences between eu-LISA Regulation and the Regulations governing the systems the Agency is responsible for. One exception relates to the e-CODEZ Regulation, which presents certain specific features that merit attention:

(1)The e-CODEX Regulation specifies that the operational management of e-CODEX shall take place in Tallinn 76 . While this provision is intended to safeguard the respect of the principle of independence of the judiciary, it is not entirely coherent with eu-LISA’s organisational set-up, as the relevant IT facilities and staff are based in Strasbourg.

(2)The e-CODEX PMB is institutionalised as a permanent body with members that are not part of the MB. This arrangement differs from the other system’s PMB and may require additional efforts to align priorities and decisions between the MB and the e-CODEX PMB. Moreover, as e-CODEX PMB members are not part of the MB, they do not always have an overarching view of the Agency’s management and priorities.  

(3)The e-CODEX Regulation specifies that resources dedicated to e-CODEX should focus exclusively on this system. This creates constraints for the Agency’s overall resource management. In practice, this approach may limit eu-LISA’s capacity to mobilise additional specialised expertise for the management of e-CODEX when needed, thereby reducing flexibility and potentially affecting the efficiency of its operational management.

4.2.How did the EU intervention make a difference and to whom?

This section assesses the extent to which the Agency’s mandate has brought added value and benefits compared with those that Member States would have achieved alone.

Could the objectives set up in eu-LISA Regulation have been achieved without EU intervention?

The evaluation has found that the Agency is playing a crucial role in contributing to high levels of security within the area of freedom, security and justice and that such degree of security could have not been achieved without a central EU Agency. The systems that are centrally managed by eu-LISA are the precondition for the existence of the Schengen area and are needed to ensure that the external Schengen borders, as well as the free movement of citizens and objects within Schengen, all take place within a uniformly secure, lawful and well-coordinated manner.

The evaluation confirmed that the development and management of systems only at the national level by Member States and/or by Schengen Associated Countries would be a considerably less cost-effective approach, and would highly increase the need for coordination activities between Member States/Schengen Associated Countries, which is prone to misunderstandings and likely creating additional administrative and financial burden. In addition, the level of information exchanges between competent authorities of the Member States would have likely been lower, encroaching on Member States’ cooperation. This would lead to a more fragmented IT landscape and more prone to security challenges, increasing the risks of unlawful Schengen border crossings or complicating the search for criminals or missed persons within the Schengen area.

However, the evaluation underlined the need for the Agency to remain flexible and adaptive, in order to continue to fulfil its mandate in accordance with the objectives underpinning its existence. This would include, amongst others, the strengthening of the tasks in research monitoring and development function to contribute to the development of IT-based solutions in the area of freedom, security and justice. In this respect, eu-LISA should not carry out pure research activities, but rather strengthen its capacity to monitor technological and research developments, establishing a robust technological watch function, which will allow them to identify emerging solution that may be relevant for the systems it manages. This would enable the Agency to ensure that innovative technologies are not only identified at an early stage, but also assessed in terms of their applicability and integrated swiftly and efficiently into the systems where relevant. In parallel, enhancing cooperation with third-country actors and international organisations, as well as further refining its internal organisation, will allow eu-LISA to remain efficient and effective.

How has the new mandate of the Agency contributed to supporting Member States in developing the information systems at national level?

Concerning the effects of Article 16 of the 2018 eu-LISA Regulation, concerning the advice, requests of ad hoc support, or the development of IT components to assist Member States in implementing technical aspects of obligations deriving from Union law, the findings of the evaluation can be read in the relevance section. In particular, the evaluation found that the the requesting procedure for IT-components under Article 16(4) of the current Regulation is rather lengthy and complicated, which has led to the Article not being invoked. In addition, as to the hand-in of the reports, no cases of support for testing procedures on the MS-level were known, also pointing to significant room for improvement.

4.3.Is the intervention still relevant?

This evaluation criterion assessed the relationship between the needs and problems at the time of the adoption of the new mandate of the Agency and during its implementation, as well as the relationship between the current and future needs and problems in the EU.

To what extent did the scope of the mandate and the objectives of eu-LISA Regulation remain relevant over the implementation period?

The evaluation found that, overall, the scope of the mandate and the objectives of eu-LISA Regulation (see the effectiveness section on the definitions of general and specific objectives) remained relevant. Throughout the evaluation period, the systems managed by eu-LISA functioned appropriately in view of political, social and other external challenges. However, the evaluation showed that the operational management of the systems could be improved to maintain a high relevance from an operational perspective. This concerned, for example, the extent to which the Agency was able to adapt its IT systems and operations to the technological progress. With particular regard to the development of new systems, the evaluation showed that the Agency’s mandate also stayed relevant across the implementation period.

While certain stakeholders expressed less satisfaction in this regard compared to the operational management of existing systems, the Agency’s mandate remains nonetheless relevant. In particular with regard to existing security and migratory challenges, the operational performance of the Agency is a key success factor for the Agency to remain relevant in the future.

How did the objectives of eu-LISA Regulation correspond to wider EU policy goals and priorities?

The wider EU policy goals and priorities considered by the evaluation concern those of the Union in the area of freedom, security and justice. The evaluation found that the general, specific and operational policy objectives of eu-LISA Regulation are aligned with the wider EU policy goals and priorities in the area of freedom, security and justice.

The eu-LISA Regulation and the specific establishing Regulations enable the Agency to develop and manage operational IT systems aimed at delivering on the specific objectives in these three areas of the area of freedom, security and justice, namely, border checks, asylum and immigration, police cooperation, and judicial cooperation in civil and criminal matters. While the IT systems concerning border checks, asylum and immigration were already mentioned in the 2018 eu-LISA Regulation, the systems concerning judicial and police cooperation were added by later amendments in 2019, 2022 and 2023 triggered through the systems’ establishing Regulations.

The evaluation confirmed that the objectives of the eu-LISA Regulation positively enhance overarching goals of the EU in the area of freedom, security and justice.

To what extent has eu-LISA adapted to the new needs and challenges, in particular to the technological or scientific progress since the adoption of eu-LISA Regulation?

Since the adoption of the new eu-LISA Regulation in 2018, new needs and challenges emerged for the Agency. This included new technological advancements such as cloud computing, artificial intelligence, improved cybersecurity measures, or blockchain technology which the Agency has not integrated into its operations. On a broader scale, the Agency was also impacted by the increase in global migration flows that affected Member States.

The evaluation found that, as concerns the technological advancements, the Agency has adapted to the new technological environment to a moderate extent. On the other hand however, eu-LISA was able to deal to a large extent with the challenges to its activities as a follow up to the increase in global migration flows. For this purpose, the evaluation assessed the number of incidents for the respective large-scale IT systems. In regard to the number of incidents, SIS maintained an availability rate of 99.9% in 2022 with a response rate of 99.0%. The same stable picture holds for other large-scale IT systems that are linked to migration flows.

Overall, the evaluation underlined the importance of intensifying research activities within the Agency and pilot testing to be able to address technological or scientific progress. New needs and challenges identified during consultations include AI for identification, biometric data, more services for judiciary, cloud services, cybersecurity, agile workflow, and improvement regarding communication and test environments. This holds for potential challenges for DubliNet, Eurodac, SIS, and VIS, as these systems must be kept up to date as well as deal with increasing flows of migration.

Has the nature of the work of eu-LISA experienced such a change that a change of mandate/ objectives is needed?

Through the implementation of the revised Regulation since 2018, the Agency experienced a pressure to not only deliver on the operational management of existing systems but to develop new systems. The evaluation underlined specific challenges with regard to parts and the flexibility of the mandate (see effectiveness section).

The evaluation looked into reasons both in favour and against a change in the mandate of the Agency. An update of the mandate could enhance the Agency's adaptability to evolving needs and technological advancements, potentially fostering innovation and efficiency. However, a change in the mandate could bring disruptions to ongoing operations at a moment where stability is needed to fulfill current objectives. In addition, changes to the current mandate would introduce additional complexities and administrative burdens, potentially diverting resources and attention away from critical operational priorities. Thus, a careful consideration and evaluation of the ‘pro and cons’ arguments are essential in determining the best course of action that balances adaptability with operational continuity.

The evaluation also noted that the Agency has recognised and acknowledged that the change of the nature of its work has brought about challenges and, thus, proactively initiated a transformation process.

Has eu-LISA’s mandate provided the necessary flexibility to address unexpected challenges?

The evaluation found that the overall mandate did not leave significant flexibility for the Agency to be entrusted with new tasks without amending the legal framework.

While Article 1(5) and Article 9 of eu-LISA Regulation explicitly refer to the possibility for eu-LISA to be entrusted with the preparation, development and operational management of additional large-scale IT systems not yet referred to in the Regulation, this is only possible if ‘provided by relevant Union legal acts governing those systems’ (Article 1.5 of eu-LISA Regulation). In practice, this triggers an amendment of eu-LISA Regulation, a procedure which has taken time based on the based on the timeline of an ordinary legislative procedure (around one to two years).

Article 16(4) could present another potential source of flexibility, as it provides the possibility for a group of at least five Member State to entrust the Agency with developing, managing or hosting a common IT component. This provision has not been used, based on its limited scope and rigid procedures. Therefore, the evaluation found that this provision has not contributed to the flexibility of the Agency in taking on additional tasks.

The evaluation also found limitations relating to the flexibility in implementing existing tasks. In particular, the evaluation pointed to the to the legal framework of the establishing Regulations of the individual IT systems as highly prescriptive with regard to the technical implementation of the IT systems, impacting the development and testing of the IT systems as well as the choice of potential IT solutions. While these separate Regulations are not the focus of this evaluation, these are relevant to consider to the extent to which they impact on eu-LISA’s activities and may interact with eu-LISA Regulation.

For example, the establishing EES Regulation specified that the system must go live in all Member States at the same time. 77  As a consequence, eu-LISA was not able to test the system in a beta version, for example, at a small number of selected border points, which is common practice in comparable projects. However, it must be noted that in 2025, the European Parliament and the Council adopted Regulation (EU) 2025/1534 78 enabling a progressive start of operations of the EES, which allows Member States to start operating the system gradually.

In addition, there are requirements in establishing Regulations of IT systems that the systems should be hosted by eu-LISA in its technical sites (e.g. Article 7.2 of the EES Regulation 79 ; Article 8 of the amending Regulation 80 ). As a consequence, eu-LISA could not consider the use of cloud services, which could be more cost efficient and would save server capacity in eu-LISA’s technical site, decreasing pressure on eu-LISA’s data centres.

With regard to specific unexpected challenges, the evaluation found that the Agency’s (in-) ability to directly and effectively respond to the Ukraine refugee crisis within the range of its core expertise – IT development – is in large part due to the limited flexibility of its mandate: 1) In a time of crisis, eu-LISA’s mandate was not flexible enough to support within the range of its core expertise; and 2) the processes governing potentially amending eu-LISA’s mandate are not swift enough in order to enable the Agency to provide operational support. As a result, despite eu-LISA being the IT agency of the EU, it was not able to effectively respond to one of the most challenging crises and security threats in recent history.

To what extent has the Agency been able to undertake additional tasks established in legal acts adopted after the entry into force of eu-LISA Regulation? To what extent does the Agency’s mandate enable eu-LISA to undertake new responsibilities?

As regards additional tasks eu-LISA established outside of the 2018 eu-LISA Regulation, the major block of additional tasks was delegated to eu-LISA by the establishment of new large-scale IT system Regulations, in particular for ECRIS-TCN (in 2019), the new e-CODEX Regulation (in 2022) and the adoption of the JITs CP Regulation (in 2023). Within these Regulations, eu-LISA was transferred the responsibility for the development and operational management of the systems, as well as related tasks (such as e.g. ensuring data security, training on use, providing support services, etc.). The situation with the e-CODEX Regulation was different, as it was an already established and operable system, managed by a consortium of Member States and organisations. During the evaluation period, the takeover by eu-LISA was completed. eu-LISA will continue to further develop the system, including through technical updates and the integration of new components.

In addition to the new systems’ Regulations, Regulation (EU) 2021/1134 on VIS and the Regulations on the renewed SIS from November 2018 81 , as well as the 2019 Interoperability Regulation (Regulation (EU) 2019/817), adding new functionalities to existing systems, brought about significant changes in the form of additional tasks of the Agency. These tasks include in particular the preparation and development of the SIS’s and VIS’s new components and adding of additional access rights, as well as monitoring of research and development efforts with regard to interoperability components such as the Common Identity Repository (CIR), Common Repository for Reporting and Statistics (CRRS), European Search Portal (ESP), Multiple Identity Detector (MID) and shared Biometric Matching Service (sBMS).

The evaluation found that, overall, the Agency has been able to successfully take on and manage the growing number and variety of tasks assigned to it. The Agency has established additional governance bodies (such as AGs and PMBs) concerned with preparing and working out specifications of the new systems and has slowly and steadily progressed towards the implementation of the systems and of the interoperability components, for which a separate AG has been established. As for other IT systems, external contractors have been included, ensuring that all necessary technical skills are represented to implement the tasks. In particular, the evaluation found that the work in the governance bodies, eu-LISA’s professional and well-developed technical environment, as well as the Agency’s secure environment were major assets of the Agency.

However, compared to the work on already operational systems, the evaluation found that the Agency need stronger management of external contractors, better planning and project management, as well as more comprehensive project and technical documentation in order to carry out these tasks successfully.

However, challenges in relation to new tasks did not only stem from eu-LISA Regulation itself: although it may be possible undertake new responsibilities from a legal perspective, the evaluation found that the extent to which this is possible also depends on the practical constraints of the Agency concerning its operation (budget, positions, prioritisation of tasks, available technology, timeframe etc.). Hence, undertaking new responsibilities successfully is a matter of aligning legislation and its practical implementation.

To what extent would the Agency be capable to play a role in the governance of the systems from a business perspective?

The expansion of the Agency’s mandate, together with the increasing scope and complexity of its tasks, has required eu-LISA to gradually evolve from a service provider into an organisation that addresses the broader business dimensions of large-scale IT systems, including their development, implementation, and management in the area of freedom, security and justice.

The evaluation found that under the current governance structure, the conduct of business analysis could benefit from further strengthening. Business analysis encompasses both the software development perspective of IT systems and the practical implementation of new systems at national level, taking into account the diverse conditions and capabilities across Member States. At present, the Management Board’s responsibilities do not extend to certain business-related issues – such as staffing and financial resources linked to system development, monitoring, or workflow management – which suggests that additional consideration in this area could be beneficial.

In addition, the evaluation noted that the limited scope of business analysis also affects the monitoring of tasks performed by external contractors. Given the Agency’s significant reliance on contractors, it is important for eu-LISA to maintain strong oversight of the delivery of their work. While contractors carry out essential functions, their participation in the meetings of governance bodies has so far been limited, which may reduce opportunities for alignment and information exchange.

Overall, the evaluation suggests that, in light of the Agency’s expanded mandate, further reflection on business-related aspects of development projects would be valuable. A stronger business-oriented approach could help ensure that governance structures are fully aligned with the Agency’s evolving objectives and scope, thereby reinforcing control over operations and development processes.

Is the current mandate and organisation of the Agency correctly reflecting the need to respect the principle of independence of the judiciary?

eu-LISA Regulation states in its Article 19 that the Management Board shall “ensure that all decisions and actions of the Agency affecting large-scale IT systems in the area of freedom, security and justice respect to the principle of independence of the judiciary”. 82  

The evaluation analysed instances related to eu-LISA’s work where this principle could be at risk or was compromised and found that eu-LISA has complied this principle. The organisational setup of the Agency favours this principle of independence: the Agency's current structure, encompassing both management and governance as well as operational functions, ensures the judiciary's independence is upheld from a broad perspective as the Agency deals with the technical architecture and infrastructure of large-scale IT systems and does not interfere with transmitted data within the systems. In line with this principle, eu-LISA’s staff does not have access to the operational data contained in the systems. In addition, eu-LISA implemented security measures such as security clearance of staff and separating the individual systems on a technical level. In the context of e-CODEX, Article 14 (2) of the establishing Regulation restricts the Agency to only commit resources for the IT system in its entirety. The Agency dedicated five staff to work exclusively on e-CODEX topics to further respect the principle of independence of the judiciary. 

5.1.Conclusions

Overall, despite a complex operational and policy environment—marked by accelerating digitalisation, rising security threats, and geopolitical instability—this evaluation concludes with a broadly positive assessment of the performance, relevance, coherence, efficiency, and EU added value of eu-LISA. The Agency has proven indispensable for the implementation and operational management of large-scale IT systems in the area of freedom, security and justice, supporting Member States and EU institutions in delivering coordinated, secure, and effective digital infrastructure.

The eu-LISA Regulation remains highly relevant to addressing current and future challenges. The Agency’s importance has increased in light of developments such as increased mobility of EU citizens and third-country nationals, rapid technological advancements heightened cyber-attacks risks, and ongoing geopolitical tensions, particularly those linked to the Russian war of aggression against Ukraine and the conflicts in the Middle East. The digitalisation of the area of freedom, security and justice constitutes a major objective of the European Union, and it is at the top of EU’s policy and political agendas, requiring a common and coordinated response by all Member States.

The objectives of the eu-LISA Regulation could have not been achieved sufficiently by Member States acting alone. The current mandate of eu-LISA, as introduced by Regulation (EU) 2018/1726, plays a pivotal role in advancing the Agency’s mission to “contribute to a high-level security within the area of freedom, security and justice, including by facilitating border management and law enforcement”. By centralising the operational management of multiple large-scale IT systems, eu-LISA not only supports Member States in their implementation efforts but also leverages significant economies of scale, fostering greater efficiency and consistency across the Union. The evaluation concludes that, while there is room for improvement in the development of new systems, the absence of an EU level Agency like eu-LISA would lead to a drastically more fragment IT landscape, compromising the security and interoperability essential to the justice and home affairs domains. Accordingly, the evaluation confirms the EU added value of the Regulation.

The current eu-LISA Regulation is to a large extent legally coherent both internally and externally. Only a limited number of challenges have been identified in relation to its coherence with other legal acts, most notably concerning its interaction with the e-CODEX Regulation. As detailed in Section 4.1.3, the e-CODEX Regulation imposes specific operational constraints -such as mandating that is management must be conducted exclusively in Tallinn, and restricting the resources provided for the e-CODEC to be committed in their entirety to that system, in order to provide for the respect of the principle of the independence of the judiciary. These provisions hamper the efficient operational management of the system by eu-LISA, showing areas where regulatory adjustments could enhance administrative flexibility and operational synergies.

No challenges relating to the internal coherence of the eu-LISA Regulation were identified. The evaluation analysed the extent to which the current structure of the eu-LISA Management Board is coherent with safeguarding the respect of the principle of independence of the judiciary. eu-LISA is a technical Agency that is responsible for the management and development of IT systems, which means that it deals with the technical architecture and infrastructure, but eu-LISA’s staff does not have access to the data transmitted via the systems they manage. While there is still a risk that there might be (criminal) incidents of infringements, the Agency has taken preventative measures. The evaluation concluded that, at this stage, it is not expected that the current organisational and governance set-up of eu-LISA would interfere with the respect of the principle of the independence of the judiciary.

The assessment of eu-LISA’s effectiveness is mixed. While eu-LISA performs well in the management of existing large-scale IT systems, with positive stakeholder feedback on service continuity and incident response, there is an important scope for improvement concerning the development of new systems.

The effectiveness of the eu-LISA Regulation in achieving its objectives differs between tasks:

-The operational management of existing systems is generally very effective, and it has been identified as a strength.

-The development of new systems is less effective, with the Agency encountering challenges related to, for example, contractor management, internal coordination, and resource allocation. These challenges have resulted in significant delays, notably in relation to the EES, ETIAS, ECRIS-TCN, and revised VIS.

-The effectiveness of the Agency in conducting tasks that are not related to the systems is mixed, with positive outputs in research monitoring, but the limited use of pilot testing and ad hoc support to the European Commission and Member States, as provided in Articles 15 and 16 of the current eu-LISA Regulation.

With its achievements relating to the development, management and operation of IT systems as well as to monitoring of research, piloting and testing, eu-LISA was able to contribute to a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice, the development of IT-based solutions in the area of freedom, security and justice as well as the implementation of evidence-based policy-making in this field. Thereby, the Agency contributed to a high-level security within the area of freedom, security and justice. However, there is room for improvement based on the shortcomings relating to the development of the systems.

In regard to the objective of contributing to a coordinated, cost-effective and coherent IT environment at Union level in the area of freedom, security and justice, eu-LISA is, to a large-extent, effective. The management of existing IT systems has been eu-LISA’s core strength over the past years. Stakeholders’ satisfaction with the performance of the Agency is high in particular with regard to the services the Agency offers concerning the system operated by the Agency, as well as their security. Evidence suggests that there is small room for improvement with regard to operational requirements.

eu-LISA’s effectiveness with regard to the development of IT-based solutions in the ASFJ is positive but there is ample room for improvement for the Agency as a whole, in particular with regard to the Agency’s overall operating model and strategy and concerning the organisation and resource management of the Agency.

Finally, the effectiveness with regard to the implementation of evidence-based Union migration and security policy is limited. While the Agency is compiling and publishing relevant statistics and produces statistical reports, and the website of eu-LISA contains links to relevant documentation about the Agency’s and the governance bodies’ activities, the accessibility and user-friendliness of these publications is limited. This negatively impacts on the extent to which the information can be used in practice and, therefore, the transparency of the Agency’s activities overall.

The assessment of eu-LISA’s efficiency presents a nuanced picture While the Agency has demonstrated strong efficiency in the operational management of existing large-scale IT systems, there remains considerable scope for improvement in the development and delivery of new systems. Challenges in this regard largely stem from difficulties in contractor management and difficulties adhering to timelines for major projects, such as the EES and ETIAS. From an organisational and governance perspective, eu-LISA operates efficiently and continuously strives to optimise its internal processes. Nevertheless, certain structural issues have been identified that hinger the Agency’s optimal performance, for instance, the rigid unit-based set-up, oversized teams and overlapping responsibilities, particularly in stakeholder engagement.

Additionally, eu-LISA’s operations are fragmented across different locations in Tallinn, Strasbourg and Sankt Johann im Pongau. These factors have, at times, contributed to inefficiencies, notably in contract management and oversight, as well as recruitment of the Agency’s staff. The Agency’s governance framework is inherently complex and can generate substantial administrative overhead. While this complexity is to some extent justified by the need to ensure robust coordination across Member States and EU institutions concerns have been raised about the administrative burden resulting from meeting-heavy schedules. The evaluation concludes that streamlining governance mechanisms and enhancing coordination among governance bodies could generate valuable synergies, reducing organisational overhead. At the same time, eu-LISA’s excessive reliance on external contractors for technical delivery poses persistent challenges, limiting its ability to effectively oversee complex IT projects. Addressing this dependency by boosting internal knowledge, improving talent retention, and strengthening project ownership is essential for ensuring long-term organisational resilience.

eu-LISA is recognised for its competent and efficient operational management of IT systems. Stakeholders’ perception of eu-LISA’s performance in this regard is positive: in the limited number of incidences or service disruptions that have taken place, the Agency was able to swiftly implement mitigating actions to ensure the continuity of service. However, there is room for improvement to enhance efficiency through finding a strategic balance between handling support functions in-house and outsourcing them to external providers. In contacts, the efficiency of new system development is more limited. The development of the EES, for instance, was subject to multiple delays and, therefore, was ultimately not developed within the foreseen schedule. Although stakeholder feedback is not overly critical, satisfaction levels of the Agency’s efficiency in terms of system development lag behind those for operational management.

5.2.Lessons learned

The evaluation concluded that the current mandate and objectives of the Agency remain largely adequate. While the Agency has demonstrated effective performance in fulfilling several aspects of its mandate, there remain opportunities for improvements in the areas of effectiveness, efficiency and coherence.

Challenges persist, particularly concerning the development of IT systems. There is also scope for improvement in as regards the alignment of the current eu-LISA Regulation with other relevant legal instruments, cooperation with international organisations, as well as piloting, testing, and ad hoc support to Member States. These challenges are linked to a range of internal factors, including strategic planning, budget and human resources management, technology infrastructure, contractor oversight, and aspects of the Agency’s organisational and governance structure.

Taking these findings and the complex operating environment into account, the evaluation confirms that, at this stage, eu-LISA’s mandate remains appropriate. Since the adoption of the current eu-LISA Regulation, two of the critical systems, the renewed Schengen Information System and the Entry/Exit System, as well as the first component of the Interoperability framework, the shared Biometric Matching Service, while four others (ECRIS-TCN, Eurodac, ETIAS and the Communion Identity Repository) are scheduled for operational launch in 2025-2026. Although modifications to eu-LISA’s current mandate could be beneficial in some areas, any significant alterations to the mandate at this stage could divert the Agency from its core objectives and jeopardise its capacity to deliver the planned priority IT systems on time, resulting in further delays in the rollout of the Interoperability Framework.  

In particular, modifying eu-LISA's mandate by amending the current eu-LISA Regulation, in the near future, would pose significant challenges at both strategic and operational levels:

·Increased risk exposure: modifying the mandate, in particular changes to Article 16(4) of the current eu-LISA Regulation, regarding ad hoc support to Member States, could heighten the Agency’s risk exposure. While greater flexibility in this area could be desirable, the evaluation suggests that eu-LISA should completing the current on-going projects before undertaking additional tasks. Any additional tasks entrusted to eu-LISA through a modification of Article 16(4) would increase the risks of further delays in the Interoperability Project.   

·Instability and delays: introducing a revised mandate at this time could create instability given the Agency’s commitment and task to deliver four systems and one interoperability component within the next two years. A change of the mandate at this moment would likely increase the risk of delays. . 

·Budgetary constraints: an amendment to the mandate does not automatically mean additional resources. Therefore, any increase in tasks would necessitate commensurate financial support, including a new Legislative and Financial Statement, which would also increase financial instability.  

Certain weaknesses identified in the evaluation, such as the cooperation with third-countries or the use of innovative technological solutions for the operation and management of the system, including the use of cloud technology, would require a change of eu-LISA’s mandate. In contrast, a large number of other issues identified can be addressed operationally without an immediate mandate change. Therefore, while the Agency would benefit from a change in the mandate, most of the immediate shortcoming identified can be addressed through operational changes, providing the Agency with the necessary stability to deliver on the more time pressing projects. 

The evaluation finds that not amending the mandate at this stage could for now better serve eu-LISA to deliver effectively and efficiently on its ongoing tasks. Therefore, eu-LISA should prioritise organisational stability and strategic operational improvement over immediate changes to its mandate. The challenges identified by the evaluation that may require a mandate revision can be addressed once eu-LISA has completed its most pressing tasks such as delivering the planned IT systems and implementing the interoperability architecture.  

Based on this conclusion, the evaluation includes a list of 54 recommendations (under the heading ‘global assessment of the areas of attention’) have been prepared. The priority order of the recommendations in the list is based on (1) their urgency; (2) their feasibility; and (3) the evaluation criteria. Considering the evaluation’s finding that mandate changes could create further risks and impediments for eu-LISA in the current context, mandate-related recommendations have been de-prioritised in the short term.

On this basis, the evaluation has identified the following 11 recommendations, to be implemented by eu-LISA, as the most important, as they can be implemented without changes to the current eu-LISA mandate and will contribute directly to organisational stability and the timely delivery of the Agency’s tasks:

-Develop a Target Operating Model (TOM) to help eu-LISA operate effectively while managing risks, making decisions quickly, and ensuring clarity in responsibilities, in particular, in case of incidents or emergencies.

-Develop and implement a comprehensive medium-term HR strategy.

-Establish and implement a medium to long-term technology strategy.

-Implement a multiannual portfolio management approach.

-Improve resource planning functionalities, e.g. through establishing processes and using appropriate tools to address incomplete view of staff allocation and steering.

-Deploy an integrated tool for the development cycle (such as JIRA) across eu-LISA and contractors to streamline project management and collaboration, minimising disruptions, optimising workflows, and ensuring relevance.

-Establish and implement a medium-term sourcing and outsourcing strategy.

-Strengthen existing capacities concerning in-house legal services.

-Update the service catalogue with full descriptions of services and define clear roles and responsibilities for service owners and managers.

-Improve visibility about current resource allocation, project status, processes, bottlenecks, shortages, in order to enable more informed discussion about the prioritisation of projects, also at political level.

-Decrease dependency from external contractors, while increasing project ownership and knowledge retention. 

1.Lead DG, Decide reference and Work Programme reference

DG Migration and Home Affairs (DG HOME) is the lead DG. There is no reference to the evaluation and the review in the Commission Work Programme 2024.

2.Derogations granted and justification

In conducting the evaluation, no exceptions from the usual procedural requirements described in the Better Regulation Guidelines were required. Public consultation did not take place as eu-LISA’s core tasks are highly technical, sensitive and of confidential nature and it was concluded that a public consultation would not bring specific added value. However, as part of the evaluation, the public was given the possibility to respond to the call for evidence. Furthermore, extensive targeted consultation of relevant stakeholders was planned and took place during the evaluation process (as described in Annex V).

3.Organisation and timing

The Terms of Reference (ToR) for engaging an external contractor to carry out the study to support the evaluation of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) were drawn up in June 2023. A request for service was issued on 3 August 2023, and a contractor selected by an evaluation committee consisting of staff from DG HOME in October 2023. The kick-off meeting for the study took place on 6 November 2023 and the final report was submitted on 14 June 2024.

In line with the Better Regulation Guidelines, an Inter-Service Group (ISG) was set up within the Commission to accompany the evaluation process. Several Directorates-General (DGs) within the Commission were invited to nominate representatives to the ISG. DG HOME chaired the ISG meetings. The ISG was regularly consulted over the course of the evaluation process, typically in conjunction with the submission of specific draft reports by the contractor responsible for carrying out the external study. These consultations took place both in regular meetings, and via written consultations. The following list provides an overview of the ISG’s work over the course of the evaluation and review process:

-The ISG was convened for the first time on 19 July 2023 in order to receive a presentation of the evaluation of eu-LISA’s Regulation and to discuss and provide feedback on the draft ToR prepared for the external contractor. The meeting also served as an opportunity to discuss and comment on the call for evidence and the draft consultation strategy, which e.g. described how the external contractor should carry out targeted consultations with relevant stakeholders in the context of the study, in cooperation with and under the supervision of DG HOME and the ISG. The participants were invited to provide additional written feedback after the meeting.

-On 1 December 2023, the ISG received the draft Inception Report for written comments and subsequent approval. On the basis of the ISG’s written feedback, the draft Inception Report was further revised by the external contractor. 

-On 1 February 2024, the ISG received the draft Interim Report for written comments and subsequent approval. The Interim meeting with the ISG and the external contractor took place on 13 February. On the basis of the ISG’s written feedback, the draft Interim Report was further revised by the external contractor.

-The Commission accepted the Final Report on 10 July 2024.

-The ISG was consulted during the drafting of this Staff Working Document.

In line with the Better Regulation Guidelines, an Inter-Service Group (ISG) was set up within the Commission to accompany the evaluation process. In addition, an extended steering group was set-up, composed by the representatives of Belgium and Estonia in the eu-LISA Management Board, and tasked with providing advice to the Commission during the evaluation.

4.Consultation of the Regulatory Scrutiny Board

Not applicable, as this evaluation was not subject to consultation and scrutiny of the Regulatory Scrutiny Board.

5.Evidence used together with sources and any issues regarding its quality

The evaluation and the review primarily drew on the supporting study entitled “Study to support the evaluation of Regulation (EU) 2018/1726 of the European Parliament and of the Council of 14 November 2018 on eu-LISA”, commissioned by DG HOME in August 2023 and carried out by the external contractor (EY). The objectives of the study were to assess the performance of eu-LISA in relation to objectives, mandate, locations and tasks with a view of identifying potential areas for improvements. The study assessed the relevance, effectiveness, efficiency, coherence, and EU added value of the eu-LISA Regulation. Beside an in-depth review of relevant documentation, including legal documents, policy and strategic documents, programming and internal business documents, Agency’s outputs and publications, financial and other documents. the study also relied on extensive targeted consultations in the form of surveys and interviews with relevant stakeholders. More specifically, the external contractor conducted 3 surveys targeting Member States, and EU Agencies and bodies, with 353 replies. EY also conducted 111 interviews with stakeholders.

During the process, the Commission provided updates and consulted Member States in the eu-LISA Management Board meetings.

6.Use of external expertise

Apart from the study carried out by the external contractor, no other groups were consulted.

Additional information on the stakeholder consultation is provided in Annex V.

In line with the ToR, the work plan was structured into four Tasks: (i) Preparation, (ii) Fieldwork, (iii) Analysis and (iv) Finalisation of the evaluation. The following figure presents an overview of these tasks, their main objectives, activities and outputs.

Figure 1 – Overview of Tasks, activities and outputs

During the kick-off meeting for this project, DG HOME indicated that the open public consultation initially proposed by the study team should not be carried out for this project in line with the Better Regulation Guidelines. In line with the suggestions from DG HOME, the study team devoted the resources initially foreseen for this activity to conduct an additional survey with the Advisory Group members (Activity 2.3).

Data collection strategy

As part of the assignment, we collected various types of primary and secondary data using the appropriate analysis techniques to assess each type of data and each piece of information:

·Qualitative information was coded and thoroughly cross-analysed. This involved searching for common themes, or areas where opinions differ across the data;

·Quantitative data collected was checked for quality and coherence, and structured for quantitative analysis, including: (1) Producing descriptive statistics (counts, cross-tabulations); (2) Producing graphs and charts to visualise trends over time and comparisons; and (3) Exploring the data to detect trends, for example, in relation to different stakeholders, different activities; and

·Cost data in line with the Better Regulation Toolbox (BRG), Tools #56 and #57.

We conducted the evaluation using a multi-criteria analysis, including a cost-benefit assessment and assessment of the administrative impacts under the "one-in-one-out" approach. Data undergone a critical assessment for quality, limitations, and strengths. We systematised information aligned with the evaluation matrix, which is structured according to the evaluation criteria / questions and covers all three dimensions as defined by the Commission in the ToR. We emphasised comprehensive coverage and employed triangulation of data between different stakeholders and across desk and field research. In summary, our analysis involved two steps: data collection based on the evaluation criteria and questions, and dimension-specific analysis. Triangulation iteratively verifies data, fostering nuanced answers through pattern identification and differences examination. We collected the data both via (i) desk research and (ii) field research, which is described in the following.

Desk research

Desk research was integral to our approach in evaluating the Agency’s performance. It served as a foundational data source, informing the assessment of the Regulation's implementation, eu-LISA's functioning and performance, and (future) needs related to its activities. We mapped existing information, in order to identify gaps for potential field research, and ensured that all team members had a thorough understanding of eu-LISA's mandate and activities. Our research covered various document types, detailed in the table below and extended by the data the Commission has e.g. provided through CircaBC.

Table 1 – Selected categories of sources covered by desk research

At the end of the project, the study team has gathered and analysed around 140 documents. Given the sheer number of documents, the study team prepared a well-structured approach on registering and assessing the documents in an overview Excel, thus enabling all members to easily gain access to sources relevant to them.

Field research

The consultation strategy followed a mixed-method approach (see Table below) that ensured an extensive coverage (written surveys) with targeted tools that allowed collecting more detailed information and having in-depth discussions with relevant stakeholders (interviews, focus groups and workshops). We aimed for a profound integration of relevant stakeholders, which was crucial to ensure access to relevant information (validity of findings) and a high level of credibility of the results (acceptance). The approach was based on a careful analysis of the objectives and context of this evaluation and ensured an active engagement of both eu-LISA and relevant external stakeholders at the EU and national levels.

While the burden on stakeholders was kept to a minimum and engaging each stakeholder only to the extent necessary and with the suitable tools, our strategy involved a wide range of stakeholders within the Agency’s units (including e.g. the advisory groups), systems’ users, other relevant EU institutions, bodies, offices and agencies and different political stakeholders.

Table 2 – Consultation tools used

Quality of information and data collected

The data collected for the completion of this evaluation included:

-Documents analysed as part of desk research;

-Exploratory interviews with the European Commission;

-(Group) interviews with eu-LISA management, staff and governance bodies, some conducted during the visits to Tallinn and Strasbourg that also included tours of the premises;

-Interviews and focus groups with private sector actors;

-Three targeted written surveys for Member States representatives, users of the IT systems and data protection authorities; and

-Six SWOT Workshops with national experts of individual systems and interoperability.

Concerning the document review, around 140 documents were analysed (public and not public). These documents include legal provisions applicable to eu-LISA and those regulating other EU large-scale IT systems, technical documentation, system reports, documents related to projects and studies, strategic documents, governance-, or human resources related information, among others. The quality of these documents was satisfying and met our expectations, contributing to a sound basis for the evaluation relating to all three dimensions covered. Nevertheless, the information was not fully complete. In particular, the Study Team did not receive documentation regarding information requested. For around 10 (types of) documents, including, among others, statistics on data protection, exit-survey outcomes with former staff, training evaluation forms, statistics on job advertisements (e.g. numbers of application, number of rejections, etc.), documentation on the volume of outsourcing, the Study Team had not had access. A list of documents received from eu-LISA is presented in the Annex (section Error! Reference source not found. ).

The interviews conducted have provided insights from various perspectives. Overall, the vast majority of interviewees were very responsive in the scheduling of the interviews and were forthcoming with providing helpful and constructive information, openly sharing their experiences and opinions. Several persons indeed highlighted the importance of the evaluation from their perspective. As concerns the interviewees with eu-LISA, some of them tended to focus on their personal challenges and making suggestions for future improvements. While this is highly relevant, efforts have been put in ensuring that the interviews were balanced in (1) covering not only challenges, but also strengths relating to the current performance of eu-LISA; and (2) focusing on the past five years (retrospective view).

With the completion of this tasks, we conducted interviews with 106 different stakeholders. This includes eu-LISA management staff, eu-LISA governance bodies, national authorities, contractors, sea-, land-, and air carrier, as well as other EU Agencies and bodies. The interviews and focus group conducted were scheduled for up to one hour each. The content derived from the interviews as well as specific information of interviewees was encrypted to ensure complete anonymity of participants.

Apart from the insights received from the virtual and on-site interviews, the visits of eu-LISA sites also provided highly valuable insights for the evaluation based on well-organised tours of the premises. In particular, they contributed significantly to the understanding of the work of the Agency, including the Agency’s tasks as well as work culture and atmosphere at the different sites.

The targeted written surveys were open from the end of January 2024 until 29 February 2024. In total, the Study Team has received 353 completed responses, including 175 for the survey with Advisory Group members, 132 for the survey with users and 46 for the survey with data protection authorities as presented in section 0 . The quality of the responses received is adequate considering that 41% of all respondents completed the full survey and 57% of all respondents across the three surveys completed the survey to at least 50%. Table below shows the statistical outcomes of the three individual surveys.

Table 3 – Overview of reponses to the targeted written surveys

Source: External Study.

Participants also used the open questions within the surveys to provide additional qualitative input, including e.g. valuable insights relating to areas for improvement (e.g. tasks outsourced, governance structure or the operational management improvements). In addition, numerous participants voluntarily flagged their availability for a follow-up interview. Throughout the process, the Study Team has been constantly monitoring response rates and quality of the responses and implemented the planned mitigation actions (e.g. sending out reminders) as needed.

As an additional part of the data collection exercise, we conducted six different SWOT Workshops on the different large-scale IT systems as well as on interoperability between 21 February 2024 and 7 March 2024. 83 These workshops were prepared for national users and members of the respective Advisory Groups of the different systems who provided us information about the strengths, weaknesses, opportunities and threats of the specific systems from their national perspectives. Across all SWOT workshops, participants represented a wide range of Member States and Schengen Associated Countries. An average session was joined by eight diverse participants. This number allowed to obtain in-depth details about system-related aspects. For collection information in an interactive manner, the Study Team prepared session-specific Mural boards on which inputs, thoughts and ideas were collected. Once the strengths, opportunities, weaknesses, and threats were collected and ranked by prioritisation, participants derived potential recommendations and assessed their individual impact and feasibility.

Description of problems encountered and mitigation actions

The identification of potential risks and/or problems encountered, and the development of mitigation actions is an integral part of the quality control measures. The following table presents the problems encountered.

Table 1 – Problems encountered and mitigation actions

The following tables present the evaluation matrix, consisting of separate tables for each evaluation criterion.

Relevance

Table 1- Relevance – Evaluation matrix

Effectiveness

Table 2 - Effectiveness – Evaluation matrix

Efficiency

Table 3 - Efficiency – Evaluation matrix

Coherence

Table 4 - Coherence – evaluation matrix

EU Added Value

Table 5 – EU Added Value – Evaluation matrix

This Annex provides a concise summary of all consultation activities conducted in the context of the “Study to support the evaluation of Regulation (EU) 2018/1726 on eu-LISA”.

Specifically, the present synopsis report offers an overview of the consultation process, including the stakeholder clusters targeted and key topics addressed through each consultation activity. Moreover, it provides a concise summary of the main results of each consultation activity, aggregating relevant information as appropriate. An overview on the methodology and analytical models and tools used.

Call for evidence

The feedbacks received on the call for evidence differed widely in their thematic orientation, length and attention to detail.

Stakeholders from within or representing the interests of the airlines particularly focused their feedback on the implementation of the EES and ETIAS systems. The professional organisations expressed support for the objectives of the EES and ETIAS systems but raised concerns about the lack of effective preparedness by eu-LISA, which could risk EU connectivity, hamper the passenger experience, and disrupt industry operations. They emphasized the need for operational readiness before implementation and offered to work closely with eu-LISA for smooth system implementation. The airlines themselves highlighted the need for eu-LISA to allocate well-qualified staff, establish mechanisms to address inefficiencies, and streamline implementation processes. They recommended improving communication methods and project management to ensure successful IT system operations.

Feedback from the public sector, represented only by the Ministry of the Interior of Estonia, focused on potential amendments to the Agency's current legal basis. The recommendations were not included in the feedback itself but in an additional attached document, which included an analysis prepared for the Ministry. 84

Feedback received from the EU and non-EU citizens covered a wide range of different topics, including (i) the good performance of eu-LISA in managing systems like VIS, SIS, and Eurodac, (ii) suggestions that the Agency's activities might become insufficient given the current migration and security challenges, (iii) the need for improved project management and better stakeholder integration, and (vi) recommendations for enhanced operational readiness to address the challenges faced by eu-LISA in managing the large-scale IT systems.

The feedback was noted within the context of the study but did not provide significant additional inputs for the study team. The main findings were rather derived from the other consultation methods outlined above, also including many of the stakeholder groups who have answered to the call for evidence.

Consultation methods and stakeholders consulted

The consultation activities performed in the context of the present Evaluation Study include exploratory and expert interviews (individual and in focus groups), workshops and a survey. Overall, consultation activities reached around 500 stakeholders, including more than 114 contributors via interviews/focus groups (including exploratory interviews), roughly 40 participants in SWOT-workshops, and 353 respondents via the online survey. The overall objective of the consultations conducted was to better understand the current functioning of eu-LISA, and where challenges arise or where the performance of the Agency could be improved with regard to its current mandate.

The table below provides an overview of the involvement of each stakeholder category into the consultation activities conducted. The stakeholder groups consulted include: officials from the European Commission; stakeholders from within eu-LISA’s staff and the Agency’s governance bodies (Management Board, Advisory Groups, Programme Management Boards); stakeholders within other EU institutions and bodies; private sector carrier organisations (air, land, sea) affected by the systems managed by eu-LISA; contractors of eu-LISA; national data protection authorities; and finally users of the large-scale IT-systems.

Table 1 - Overview on consultations conducted by stakeholder group

Interviews

Throughout the project, the Study Team performed two rounds of interviews, namely (i) six exploratory interviews with officials from the European Commission (DG HOME AND DG JUST) (ii) 108 expert interviews with stakeholders from within (i) eu-LISA staff, and the governance bodies of eu-LISA, and (ii) EU-level and national level stakeholders, comprising the European Commission (in particular DG HOME), other EU-level institutions and agencies, including Frontex, Europol, and the EPPO, as well as private sector actors, in particular contractors of eu-LISA. All interviews were semi-structured and conducted by means of interview guides that were approved by the Commission as part of the preparation of the inception report and coined towards the respective stakeholder group.

Exploratory interviews

In the inception phase of the study, the study team conducted six one-hour exploratory interviews with stakeholders from within the European Commission. This included stakeholders from, amongst others, the following units of DG HOME and DG JUST: DG HOME Directorate B (i.e. DG HOME B3, DG HOME E1) and DG JUST H4 and DG JUST A4. Most interviews were conducted virtually via MS-Teams, with one exception where the interview was held in-person.

The interviews aimed at deepening the study team’s understanding of the functioning of eu-LISA, and to finalise the evaluation framework, intervention logic and data collection instruments. Moreover, the interviews allowed the study team to collect further documentation and identify additional data sources.

The interviews followed a semi-structured approach to provide sufficient guidance, ensuring that the relevant aspects were covered while leaving sufficient flexibility to tailor each conversation to the focus area and expertise of the interviewee. To allow for the semi-structuring of the interviews, the study team prepared interview guidelines in advance of the consultations, which followed the five evaluation criteria provided in the EU Commission’s Better Regulation Guidelines 85 : Effectiveness, Efficiency, Relevance, Coherence, and EU Added Value. In addition, the study team prepared questions following-up on the recommendations provided in the previous evaluation of the eu-LISA Regulation conducted in 2016/2017, as well as on lessons learned and the way forward.

Expert interviews

During the fieldwork phase of the study, overall, 108 interviews were conducted with (i) stakeholders within eu-LISA and the governance bodies; and (ii) EU-level and national level stakeholders. Most of the interviews were conducted virtually via MS-Teams, however, the study team used the opportunity of the exploratory visits to Tallinn and Strasbourg to also conduct interviews with eu-LISA staff in-person (overall 28 interviews were conducted in-person). The main goal of these interviews was to gather detailed insights into the views of eu-LISA staff and members of the governance bodies, to better understand the current functioning of the Agency as well as existing challenges and areas where room for improvement exist.

In total 70 interviews (of the overall 108 conducted interviews, i.e. roughly 65% of all expert interviews) were conducted with stakeholders within eu-LISA and its governance bodies. Of the 70 interviews, 46 interviews (66%) were conducted with eu-LISA staff whereas 24 interviews (34%) were conducted with representatives from the governance bodies. An overview on the number of interviews per department/unit is found below in the table.

Table 2 - Stakeholders reached within eu-LISA staff

On the side of the governance bodies, the remaining 24 stakeholders contacted were either from the Management Board or from the Advisory Groups and the e-CODEX PMB. Overall, 12 stakeholders were contacted respectively for each of the two bodies, covering different Member State perspectives. As concerns the Advisory Groups and e-CODEX PMB, at least one stakeholder could be reached for each AG and for the e-CODEX PMB. Given that a maximum of two to three interviews per AG and e-CODEX PMB were initially planned and that the same stakeholder groups were also contacted via the online survey, this is a satisfactory result.

As for the exploratory interviews, the one-hour interviews were conducted following a semi-structured approach, following pre-agreed upon interview guidelines. For all interviews, questions were clustered by the three main dimensions covered by the evaluation, which were provided in the ToR. These included (i) the mandate, tasks and objectives of the Agency, (ii) the governance and location of the Agency, and (iii) the management of resources. In addition, the study team drafted specific question blocks for each stakeholder group to allow for deep-dives concerning specific topics according to the expertise of the interviewees. Before the interviews, team members were briefed to ensure familiarity with the interview guides and approach to conducting the interviews.

As concerns the remaining 38 EU and national stakeholders reached via interviews/one focus group, an overview of the stakeholders contacted can be found in the table below.

Table 3 – EU and national level stakeholders reached

For the interviews with EU Agencies and bodies as well as the contractors, interview guidelines were drafted. Similar to the above interviews, the interview guides were approved by the Commission as part of the submission of the inception report and coined towards the respective stakeholder group, covering aspects such as administrative procedures, cooperation with eu-LISA, questions about eu-LISA’s current operations as well as future needs and developments to consider. As for the other interviews, the study team’s interviewers were briefed prior to the start of the fieldwork phase to ensure compliance with the guidelines.

The one-hour focus group was conducted on 22 February 2024 via MS-Teams with 20 participants representing air (five representatives), land (one representative) and sea carriers (11 representatives), as well as three other representatives from other carrier groups. The focus group was conducted with the help of the online whiteboard tool MURAL, which allowed the participants to contribute their ideas and perceptions live during the focus group. In addition, the study team allowed for more detailed exchanges in two break-out sessions. In order to foster the discussions, the study team prepared a number of guiding questions prior to the focus group.

Online survey

A written online survey was distributed among three different stakeholder groups in order to gather information on the functioning of eu-LISA, experiences made with the large-scale IT-systems operated by the Agency, and on the Agency’s adherence to data protection standards and guidelines. Stakeholders contacted comprised: (i) Advisory Group members and the e-CODEX PMB; (ii) The current and future users of the different IT systems (i.e. national competent authorities (NCAs), such as national law enforcement authorities, border control authorities and immigration authorities as well as the judiciary); and (iii) national supervisory authorities. The surveys covered aspects relevant to these groups, containing filter questions to ensure that only relevant questions were posed to the stakeholders. The different surveys were approved by DG HOME as part of the submission of the inception report. Overall, 353 answers were recorded through the survey. An overview of the quality of responses to the targeted written surveys can be found in the main body of the report in Table .

As concerns the survey distributed among AG members and the e-CODEX PMB, the study team reached 175 stakeholders via the survey. The figure below provides an overview on the distribution of respondents within the AGs and the e-CODEX PMB. Most answers were received from members of the EES-ETIAS Advisory Group whereas the e-CODEX PMB, as to be expected due to its size, provides the smallest number of answers. It should be noted that the survey participants were able to provide multiple answers to this question, which is why the number of respondents in this specific question is higher than the overall number of stakeholders reached (175, see above).

Figure 1 - Share of AG and e-CODEX PMB survey participants by AG/PMB (n=179)

For SIS, VIS and Eurodac, the questions focused on the operation and functioning of the systems, whereas the questions for ECRIS-TCN, EES-ETIAS and e-CODEX had a more forward-looking focus on development-related aspects of eu-LISA’s work. For participants of the surveys related to the judicial systems, there were additional questions on the principle of the respect of the independence of the judiciary and the handover / takeover process of e-CODEX from a consortium to eu-LISA. For all respondents, the survey concluded with a block of questions on organisational matters and governance, as well as future challenges and needs.

Regarding the survey distributed among current and future users of the large-scale IT-systems, the study team reached overall 132 stakeholders via the survey. Even though not all EU Member States and Schengen-Associated Countries (SACs) were reached via the survey, the study team roughly reached respondents in three quarters of the countries contacted (77%, 24 of 31). An overview of the participating countries and the number of responses recorded can be found below.

Figure 2 - Share of user survey participants by country (n=90)

The overwhelming majority of participants were members of national SIRENE Bureaux (33%, 30 of 90 respondents who answered the question), followed by police officials (27%, 24 of 90), immigration, naturalisation and visa issuing authorities (9%, 8 of 90), border control (7%, 6 of 90) and asylum authorities (4%, 4 of 90). In addition, stakeholders from judicial authorities (2), other national law enforcement authorities (2), Europol (2) and the EPPO (1) were reached.

Given that the survey focused on currently operational systems, participants were asked which system they wish to provide answers for. The majority of respondents indicated that they would like to answer questions for SIS (71%, 64 of 90 respondents who answered the question). Second most answers were provided for VIS (24%, 22 of 90), followed by Eurodac (18%, 16 of 90) and lastly DubliNet (6%, 5 out of 90). It should be noted that respondents had the opportunity to provide answers for multiple systems, which is why the percentages do not add up to 100%. The questions within the survey focused on the use and satisfaction of the respondents with the IT-systems and also included additional questions on system development and the respect of the principle of the independence of the judiciary where applicable. As for the AG and e-CODEX PMB survey, the user survey concluded with a block on future challenges and needs.

Lastly, the study team distributed a survey among national supervisory authorities, which reached 46 stakeholders. Stakeholders were able to provide answers to questions concerning the General Data Protection Regulation (GDPR) 86 other legislation, as well as specific questions as concerns the data protection requirements and challenges of the single operational IT-systems (SIS, VIS, Eurodac, DubliNet). Overall, 65% (20 of 31) of contacted Member States and SACs provided at least one answer to the survey. Answers differed between one and three per country, with respondents from Germany providing most answers as a single country. In terms of expertise, most participants indicated to have core expertise for the SIS (83%, 24 of 29), followed by the VIS (65%, 19 of 29), Eurodac (45%, 13 of 29) and DubliNet (only one participant indicated to have core expertise for this system). As above, the percentage does not up add to 100% due to participants’ possibility to choose multiple answers.

SWOT-workshops

The last consultation method used in the study constituted the conducting of six SWOT workshops, aiming at identifying current Strengths, Weaknesses, Opportunities and Threats within eu-LISA’s operations. The thematic focus of the single workshops was clustered by eu-LISA’s Advisory Group structure, the operational status and the overall domain of the topics addressed in the respective workshop, leading to the following overall clustering of workshops conducted:

-ECRIS-RI, ECRIS-TCN and e-CODEX (21.02.2024)

-SIS (26.02.2024)

-VIS (28.02.2024)

-Eurodac and DubliNet (01.03.2024)

-Interoperability (04.03.2024)

-EES and ETIAS (07.03.2024)

Overall, roughly 40 stakeholders were reached within the context of the workshops, with differing participation rates depending on the workshop. Participation rates were particularly strong within Western and Middle European countries, including multiple participants from Austria, Czechia, Hungary, Germany, and the Netherlands. In addition, participants from France, Iceland, Italy, Lithuania, Luxembourg, Malta, Norway, Poland, Portugal, Slovakia, Spain, and Switzerland also participated in some of the workshops respectively.

The workshops were all conducted virtually via MS-Teams with the help of the online whiteboard tool MURAL, which served as a platform to collect inputs from the participants and structure the outputs derived. All workshops followed the same structure: First, the study team provided a short overview of the contents of the study, sharing a PowerPoint presentation with participants via screen share. Following the introduction, participants received instructions on how to use MURAL and a short tour de table were held. Afterwards, participants were able to contribute to the SWOT analysis by adding notes (“post-its”) where they saw specific strengths, weaknesses, opportunities and threats in a number of different overall categories such as technology, user perspective, governance, fundamental rights, processes, or other aspects. Lastly, the notes rated most important by participants were used to derive recommendations on how eu-LISA could further foster its strengths and upcoming opportunities as well as how to mitigate the effects of weaknesses and threats. Each workshop was concluded with a short summary by the study team.

Call for evidence

In the context of the study, the Commission has launched a call for evidence. Calls for evidence are used in the context of planning and preparing law and describe the problem to be tackled and objectives to be met, explains why EU action is needed, outlines policy options and, where applicable, describes the main features of the consultation strategy. In response to the call for evidence, private actors are able to provide feedback and recommendations on the current status of the proposed law.

The call for evidence concerning the evaluation of eu-LISA’s mandate started on 18 August 2023 and was open for roughly a month until 15 September 2023. Overall, the Commission received feedback from 15 people, of which 14 feedbacks were published and one remained unpublished. Feedback givers were able to file their feedback online via the Better Regulation Portal.

The sources of the feedback included in particular actors from the carrier sector (exclusively airlines or professional organisations representing airlines, 6 of the 15 feedbacks). In addition, a number of private persons (EU and non-EU citizens) have provided feedback, without closer defining which organisation they belong to (7 of 15 feedbacks). Lastly, one actor from the Ministry of the Interior in Estonia and a Spanish non-governmental organisation have participated in the call. In general, the feedback received covered a wide range of EU and non-EU countries, including actors from Austria, Belgium, Czechia, Estonia, Germany (2 actors), Israel, Romania, Slovakia, Spain (3 actors), Sweden, and the United States of America (2 actors).

Results of the consultation activities

The following part of the synopsis report provides a high-level overview of the main results of the consultation activities.

Interviews

Exploratory interviews

The exploratory interviews aimed at deepening the study team’s understanding of the functioning of eu-LISA, and to finalise the evaluation framework, intervention logic and data collection instruments. The main take-aways comprised:

-The rise in migration to the EU and the international character of organised crime continuously underline the importance of the Agency.

-The Agency provides considerable EU Added Value, given that single Member States’ approaches would create significant administrative and cooperation burden.

-The Agency is overall perceived as professional and effectively aligns its objectives with its operations. Its core business, the operation of the large-scale IT system, works smoothly and without major technical incidents.

-The transparency and insufficient long-term planning of budgetary and organisational procedures were considered areas for improvement.

-The development of new IT systems through the Agency as well as the current outsourcing approach were considered to be key challenges by most interviewees.

-The difference between the traditional eu-LISA systems in the area of home affairs compared to the justice systems was emphasised. Questions were raised on how effective eu-LISA will be in transferring the current e-CODEX system from a consortium to the Agency.

Expert interviews

The main findings of the expert interviews are as follows:

Dimension 1: Mandate, tasks and objectives of the Agency

Mandate and objectives:

-Most stakeholders confirmed the overall adequacy of the mandate of eu-LISA. In particular for eu-LISA staff, the mandate and objectives tended to be considered as “given” and the focus lies on fulfilling the mandate.

-The flexibility of the mandate was considered to be rather restrictive by most interviewees within eu-LISA who addressed this topic, which was stated to potentially lead to problems in terms of timely and appropriate reactions to unforeseen circumstances.

-Concerning the cooperation with international organisations (Article 43 of the eu-LISA Regulation) and third countries, some stakeholders from the Agency voiced that this provision has not been used successfully. The provision specifies that such a collaboration can only take place if provided by a Union legal act and following the approval process that involves the Management Board (MB) and the Commission. However, based on the current provision, past attempts have not been successful.

-A majority of the justice stakeholders confirmed that the mandate is adequate for the justice systems, however, they voiced concerns as regards the respect of the principle of the independence of the judiciary, given that eu-LISA might serve as a connector between the executive branch and the judiciary, thus not fully respecting the principle. In addition, justice stakeholders emphasised their wish to be more strongly represented in the Agency’s governance bodies.

Tasks:

-Article 15 on piloting and testing and Article 16(4) on ad hoc support to Member States were considered very relevant by most interviewees who answered questions on topic, based on that several requests were made by different Member States according to these Articles in the past years. However, some interviewees from eu-LISA noted that the implementation of these provisions was hindered due to the complex and time-consuming approval processes specified in the provisions. On this basis, the requesting Member States in the past changed their mind and found other solutions, as it took too long to secure the needed support by eu-LISA.

-As concerns the implementation of its tasks, a majority of stakeholders from eu-LISA and governance bodies noted that the management of existing system and communication infrastructure as well as the provision of training works relatively well. Significant room for improvement was identified with regard to the development of new systems (delays, extensive outsourcing), the monitoring of research and support function to the Member States.

Dimension 2: Governance, location and organisation of the Agency

Governance:

-Overall, the work of the governance bodies is considered important and running smoothly by a majority of stakeholders within and outside of the Agency, including the cooperation between different bodies and with eu-LISA.

-As concerns areas where room for improvement was identified, many interviewees highlighted a high administrative burden of each governance body as well as potential risks of duplications and/or contradictions as key challenges for the Agency.

-In addition, many stakeholders from the governance bodies raised that the practical work of the governance bodies is sometimes challenging due to the varying levels of expertise of its members.

-As concerns coordination with the Member States, most stakeholders confirmed well-working coordination mechanisms, with only very few exceptions highlighted.

-Finally, several interviewees from the justice area indicated that increasing challenges are expected to arise in the MB due to the increased future importance of the justice area. In particular, this has brought up questions relating to the potential inclusion of justice profiles into the so far mainly home affairs focused MB.

-The efficiency of governance could be improved through shifting from the current silo-based governance approach towards a sector-specific governance model, by clustering governance bodies, and establishing formalised channels for communication and collaboration.

Location:

-As concerns the political dimension, most stakeholders from the Agency and its governance bodies showed a general understanding for and acceptance of the decision to locate the Agency’s sites in three Member States (with the liaison office being established in Belgium, a fourth Member State).

-With regard to the operational dimension, a large majority of interviewees stated that in particular the split of the locations between Tallinn and Strasbourg, as well as between Strasbourg and St. Johann im Pongau, has de facto caused operational challenges and roadblocks over the past five years. The most frequently mentioned challenges concerning Tallinn-Strasbourg were: (1) Detachment of the corporate services from the operations; (2) Remoteness and practical reachability of both Tallinn and Strasbourg; and (3) The secluded location of the Central Unit and the rented office space in Strasbourg.

-Addressing the attractiveness dimension, the location of the Agency’s sites was criticised by some interviewees from eu-LISA as a major reason for a comparatively limited attractiveness of eu-LISA to potential applicants. Tallinn was considered unattractive due to its location far north and the cold weather conditions. In Strasbourg, the Central Unit is located at the southern outskirts of the city in a residential neighbourhood at the end of a street next to a farmland with very limited lunching/dining options or opportunities to run errands.

Organisation:

-The clarity of the organisational set-up and structure of the Agency within departments, units and sectors was considered by stakeholders within and outside of the Agency as acceptable.

-At the same time, the perceived disconnect between corporate services (in Tallinn) and operations (in Strasbourg) were viewed as problematic, creating communication and cooperation difficulties between both departments.

-Some stakeholders from the Agency criticised the lack of clear role descriptions and responsibilities, which might lead to overlaps in the work of eu-LISA.

-As concerns the size of the organisational entities, several stakeholders from eu-LISA, but also from the governance bodies voiced concerns about the current organisational structure, which may not be adequately capable of managing the number of staff allocated to each of the units and sectors. From a practical perspective, interviewees argued that the size of the units and sectors is too large to be managed efficiently.

Dimension 3: Management of human, technical, and financial resources

Human resources

-The available human resources are the Agency’s most important asset. This was also acknowledged by several interviewees within the Agency, who specifically pointed out that the collaboration with their colleagues and peers is one of their most important sources of motivation to work at eu-LISA.

-Based on the interviews conducted, there are challenges with regard to the career development within eu-LISA: With only a finite number of positions available at different levels, it seems that challenges exist for staff to ‘climb up’ the career ladder.

-With regard to the challenges in recruitment, retention and development of HR, the following factors were mentioned most frequently by interviewees within eu-LISA and in the governance bodies:

oStrategic factors: Availability of positions; Competition with private industry; Purpose and mission of the Agency (i.e. its branding); Location and facilities of the Agency;

oContractual factors: Necessity of security clearance; Necessity of short-term contracts; Salary and compensation package;

oOperational considerations: Ability to work with cutting-edge technology; Rigid ways of working.

-Most stakeholders consulted also noted that the Agency has undertaken efforts to mitigate the HR-related challenges, comprising e.g. the digitalisation of the HR process, a restructuring of the organisation of HR management, changes in the organisational development (launch of cultural transformation projects, updating of existing role descriptions), as well as improved employer branding.

-Lastly, it was noted by some stakeholders within eu-LISA that the Agency must provide a fully-fledged HR strategy in the future.

Technical resources

-According to various stakeholders from within and outside the Agency, eu-LISA is aims at adopting state-of-the-art technologies, signifying a noteworthy pursuit of innovation and operational efficiency.

-This being said, many stakeholders noted that there is untapped potential in the fields of AI-based applications and cloud technologies, notably due to restrictions in the mandate.

-Stakeholders from within the Agency and its governance bodies also identified challenges in the software development life cycle. The layered approach where testing, delivery, and application maintenance emerge as distinct layers managed by different contractors under various framework contracts, introduces complexities in coordination and communication across the different phases of the SDLC. This exacerbates the challenge of achieving seamless end-to-end delivery and needs to be addressed with specific approaches and tooling.

-In addressing these problems, eu-LISA has shifted from a more traditional waterfall model towards more agile, iterative and collaborative methods. At the heart of agile implementation at eu-LISA is the strategic formation of 'squads.' These squads are not only cross-functional teams; they are a dynamic fusion of diverse expertise and experience that operate with focused objectives, autonomy, and collective ownership.

-Given its extensive outsourcing, the Agency sometimes struggles with the retention of technical expertise and experience within eu-LISA.

Financial resources

-The financial planning and reporting are based on traditional year-by-year public budgetary processes. Within this context, many stakeholders with expertise in the area highlighted that the collaboration between the Commission (in particular DG HOME) and the Agency has positively improved over the last years, but it has still sometimes been challenging for eu-LISA to provide the Commission with the requested information in time.

-The growth of tasks of the Agency has also led to the situation of managing and updating three systems and, in parallel, develop additional new ones from scratch. It seems that appropriate planning capabilities were only established over time, which led to gaps between the actual and planned progress over time.

-With specific regard to multi-annual planning, stakeholders with specific expertise in the field pointed out that this is an area for improvement, both from a strategic perspective and the concrete tooling. Over the previous years, too much emphasis was given to the annual planning, which, in turn, led to a lack of strategic planning of resources over time. This led in the past to the Agency having asked for too many financial resources, which could, ultimately, not be used entirely.

-As concerns the Agency’s outsourcing practices, many stakeholders within the Agency and even on the side of external contractors themselves criticised aspects such as legislative predicament, the outsourcing strategy (or lack thereof), the complexity of the outsourcing approach, the contractual management, and the volume of outsourcing.

-In particular justice stakeholders noted, that in the context of e-CODEX, the respective Regulation states that “eu-LISA shall commit the resources provided (…) for the e-CODEX system in their entirety to its operation” (Article 14(2) of the e-CODEX Regulation). Hence, eu-LISA resources allocated to e-CODEX are solely focused on this specific IT system itself in order to ensure the respect of the principle of the independence of the judiciary. However, (i) currently there are five FTE working on e-CODEX which is insufficient and (ii) the legal mandate impedes efficiency as resources are rigid.

Online survey

Below, an overview of the main findings clustered by the Better Regulation Guideleines’ evaluation criteria is found.

Effectiveness

-As concerns eu-LISA’s general objective, the Agency is considered effective in contributing to a high-level of security within the AFSJ. The primary reason for this assessment is the smooth functioning of the existing IT systems. However, in view of the current room for improvement concerning the development of new IT systems, the effectiveness of the Agency could be even higher.

-As concerns eu-LISA’s effective management of the large-scale IT-systems, a majority of respondents consider eu-LISA to contribute to a coordinated, cost-effective and coherent IT environment at Union level. 24 out of 35 (69%) AG and PMB members who answered this specific question indicated in the online survey that eu-LISA’s management of the systems contributes to a ‘large’ or ‘very large’ extent to a coordinated, cost-effective and coherent IT environment.

oAs concerns the overall levels of performance, 86% (64 out of 74) of respondents from eu-LISA have said to be ‘very satisfied’ or ‘satisfied’ with the performance. Users’ assessment resembles this largely, with 85% (65 out of 75) of respondents indicating they are ‘very satisfied’ or ‘satisfied’ with the performance

oWithin the single systems, performance of SIS and VIS are rated best among AG and e-CODEX PMB members (for both roughly 50% very satisfied), whereas Eurodac only receives 40% (29 of 74) ‘very satisfied’-ratings.

oOpenness to feedback is also considered sufficient, however, the overall ratings are not as good as for system performance with only 53% of respondents (19 of 36) considering eu-LISA to be open to feedback to a ‘large’ or ‘very large’ extent.

oThe takeover of e-CODEX was handled successfully to a ‘very large’ or ‘large’ extent according to 65% (17 of 26) of stakeholders from the AGs and e-CODEX PMB consulted.

oRatings for data protection by consulted national supervisory authorities were also largely positive, with 80% (19 of 24) of respondents considering eu-LISA to be largely in accordance with the GDPR 87 , and 75% (18 of 24) of respondents considering eu-LISA to be largely in accordance with the Law Enforcement Directive. 88

Efficiency

-With regard to its efficiency, the operational management of existing IT-systems on the side of eu-LISA was rated positively by most respondents from the AGs and e-CODEX PMB consulted, with ‘very satisfied’ and ‘satisfied’ ratings accounting for at least 70% (n=74) in all categories asked (e.g. system availability, continuity of services, timeliness of service delivery, support responsiveness, quality and frequency of communication, reaction times to security issues)

-As concerns the performance of eu-LISA’s contractors, most respondents from the AGs and the e-CODEX PMB are ‘very satisfied’ or ‘satisfied’ (for each category asked at least 70% of respondents, n=38) with the performance, even though differences were seen between the categories. Respondents were comparably strongly or moderately dissatisfied with the response time (23%, 9 of 38) and limitations of impact on the operations (26%, 10 of 38) on the side of the contractors. Best scores were reached for incident management, with an overall 84% (32 of 38) of respondents being either ‘very satisfied’ or ‘satisfied’ with the work of the contractors. Even though work of the contractors was rated comparably high, it is notable that the Agency’s overall outsourcing strategy is rather rated unfit to deliver its tasks sufficiently, with 74% (44 of 60) of AG and e-CODEX PMB respondents rating the strategy successful to ‘a moderate extent’, ‘low extent’, or ‘not at all’.

-In general, the stakeholders in the AGs and e-CODEX PMB wished for eu-LISA take over more responsibilities from external contractors, with 45% of respondents (21 of 47) thinking to a ‘very large’ or ‘large’ extent that the Agency should do so.

-As concerns the development of new systems, the AG and e-CODEX PMB members participating in the survey painted a mixed picture of eu-LISA’s efficiency. Roughly 30% of respondents (14 of 68) were either ‘very satisfied’ or ‘satisfied’ with the work of eu-LISA, whereas nearly half of respondents (45%, 30 of 68) were only satisfied to a moderate extent. 25% of respondents (17 of 68) were either ‘dissatisfied’ or ‘not at all’ satisfied with the performance of the Agency.

-In terms of staff covering the required competencies to make the Agency work efficiently, most respondents (76%, 55 of 72) from the AGs and e-CODEX PMB agree that the staff qualifications available satisfy the needs of eu-LISA to ‘a very large’ or ‘large’ extent’, thus painting a positive picture on the recruitment practices of the Agency.

-Asking about the efficiency of the governance set-up of the Agency, stakeholders from the AGs and the e-CODEX PMB paint a mixed picture. Even though very low and low ratings remain an exception, roughly a third of respondents (20 of 59) consider the set-up only to a ‘moderate’ extent efficiently. Nonetheless, a majority of 34 of 59 stakeholders (58%) answered the question with a positive result (‘very large’ or ‘large’ extent). A more nuanced picture is painted by stakeholders as regards the functioning of the internal organisation, with decision-making procedures and escalation procedures highlighted by at least 30% of respondents as only ‘partially’ or ‘not at all’ appropriate.

Coherence

The internal and external coherence of the eu-LISA Regulation was assessed entirely through qualitative methods, which is why no survey item is included here.

EU Added Value

-The added value of the eu-LISA Regulation compared to what Member States could have achieved alone has clearly been confirmed in this evaluation. Without a single body such as eu-LISA responsible for several IT systems in the AFSJ, the current IT landscape would be drastically more fragmented, and it would likely be more challenging to ensure security.

-As already presented above, 69% of the surveyed AG and e-CODEX PMB indicated that eu-LISA’s management of the systems contributes to a large or very large extent to a coordinated, cost-effective and coherent IT environment at EU level in the AFSJ, which the Member States could not have achieved alone.

-In terms of support for the MS in developing the information systems on the national level, the AG and e-CODEX PMB members paint a more mixed picture, with only 45% (26 of 58) of respondents indicating that eu-LISA supports the MS to a ‘very large’ or ‘large’ extent.

Relevance

-The continued overall relevance of eu-LISA’s mandate and objectives has been confirmed in this evaluation. Indeed, the relevance of the Agency has become even more important, considering external developments such as the increased mobility of EU citizens and third-country nationals, technological developments and increased risks from cyber-attacks as well as threats from ongoing geopolitical developments and conflicts (e.g. in Ukraine and the Middle East).

-As concerns how well eu-LISA performed in view of existing challenges, 71% of respondents (27 of 38) replied that eu-LISA performs well to a ‘very large’ or ‘large’ extent, underlining the relevance of the Agency with a view to upcoming challenges.

-As concerns which challenges are considered particularly important, stakeholders from the AGs and the e-CODEX PMB pointed out the expansion of eu-LISA’s responsibilities (54%, 43 of 79), the ‘war for talent’ (45%, 36 of 79) and ensuring sufficient budget and financing (43%, 34 of 79) as ‘very important’ challenges.

-In order to keep up its current relevance, stakeholders highlighted the need to implement stronger feedback and cooperation mechanisms, increasing eu-LISA’s attractiveness as an employer, increasing the Agency’s pilot testing capabilities and adapting the current organisational structure (all considered at least to 70% ‘very important’ or ‘important’).

-To improve its current relevance, 53% of respondents from the AGs and e-CODEX PMB currently consider eu-LISA to take feedback provided into consideration to a ‘very large’ or ‘large’ extent, pointing out the need for eu-LISA strengthen its current feedback mechanisms.

SWOT-workshops

The study team conducted six SWOT workshops, focussing on the experiences eu-LISA governance body members and national users made with eu-LISA’s large-scale IT systems. The results are presented in in the following (clustered by workshop). After finalising the SWOT analysis, the participants derived recommendations based on challenges identified. An overview on the results is given in Figure at the end of this chapter.

ECRIS-RI, ECRIS-TCN & e-CODEX

Strengths

-Strong regulative framework, structure and organisation capable to deal with the complex framework

-Secure and reliable environment provided by eu-LISA

-Centrally positioned, professional and skilled support on the side of eu-LISA for the entire product

-Well-equipped premises for meetings

-Use of state-of-the-art technology and access to standardised tools and applications

Weaknesses

-Strictly defined role of eu-LISA within its mandate

-Lack of exchange of documentation documents (and general lack thereof) between different projects

-Long decision-making procedures within the Agency

-Differing knowledge between different Member States, eu-LISA, and Commission

Opportunities

-eu-LISA should contribute more strongly within the Commission comitology

-eu-LISA can become a main pillar of the e-CODEX community

-It is recommended to strengthen the role of the AGs and PMBs

-The Agency can become a stronger actor in providing standardisation and harmonise the approaches between the different Member States

-eu-LISA should also try to include new target groups of the systems

-The Agency can ensure the respect of the principle of the independence of the judiciary

Threats

-The heterogenous governance structure can lead to diffusions

-The complexity of e-CODEX might not be well understood by many members of the community

-The speed of developments is too fast for eu-LISA to cope with

-There might be a shortage of funding and resources

-The Member States might not be able to implement all necessary technical infrastructures and need more time than expected to implement all measures necessary

Schengen Information System (SIS)

Strengths

-Centralised technology and centralised solutions for the whole system

-Good service by the Service Desk

-A very supportive eu-LISA team and very pragmatic and realistically thinking staff on the ground

-Strong systems for training Member States

-Good data protection measures, conformant to legal base

Weaknesses

-Limited information for end-users / limited resources in general and extensive outsourcing on the side of eu-LISA

-Sometimes issues in availability

-It takes a long time to implement updates or changes. Overall change management is considered inefficient

-Testing performed is often too limited and strict, leading to the identification of errors when it is already too late

-Member States’ positions are not considered sufficiently by eu-LISA

-Timelines set as concerns changes in the system are too ambitious and optimistic

-Participant management for meetings need to be improved (contacting the right people, sending invitations timelier, etc.)

Opportunities

-Insourcing of more developers, to keep knowledge within the Agency

-Improving the quality and stability of technical documents

-Increase the sharing of best practices

-Take Member States’ concerns stronger into consideration and better communicate with MS upcoming changes or problems

-Build more partnerships (e.g. with Europol)

Threats

-Lack of resources can hinder the proper development and keeping of knowledge within the Agency

-Lack of proper communication might in the long-term hinder the effective and efficient cooperation with Member States

-The Agency might be too political, not considering Member States’ needs sufficiently in the future

Visa Information System (VIS)

Strengths

-The VIS is robust and reliable

-Member States can use their own applications

-The VIS provides a helpful communication tool via VIS mail

-Strong data protection framework

-Good knowledge about relevant legislation in the VIS team

Weakness

-Insufficient knowledge management (e.g. via a SharePoint)

-eu-LISA takes too much time to process requests and questions

-Internal and external communication with eu-LISA is not good; there are too many working groups increasing the administrative burden of communication

-eu-LISA should be stronger involved in drafting of legal basis

-Access rights are considered quite strict

-Too many Interface Control Documents (ICDs) are used to achieve a usable result

Opportunities

-Recommended to make documentation available via an own wiki page or something comparable

-Include more national experts and, in general, include the Member States in further decisions

-eu-LISA should get more flexibility in design (e.g. technical and functional specifications of the system)

Threats

-The dynamic environment and frequent changes of the IT environment are not considered and addressed timely

-Interoperability with other systems is not implemented successfully

-Too ambitious implementation plans might lead to delays in the long-term

-The skill shortage in IT might lead to recruitment problems for the Agency

-Pressure from the side of the Commission might lead to unfinished problems being published

Eurodac & DubliNet

Strengths

-The system is user-friendly

-Good service by the Service Desk

-The system is considered safe and efficient

-The data on Eurodac is very limited, which reduces the risk of data pollution

-The roles of the AG and PMB are highlighted very positively as supportive to Member States’ cooperation

Weaknesses

-Renewal of certificates (every two years) is labour intensive and disturbs the work flow

-New certificates are often linked to old accounts, sometimes preventing Member States to open attachments sent with new certificates

-There is no properly established change management procedure

-There is a lack of formal communication with the Member States; preparation of AG meetings’ communication (e.g. sending around relevant files) could be improved

-Changing incorrect information in Eurodac is a very complicated and lengthy process

-End-users sometimes lack knowledge on how to use the system properly

Opportunities

-Include Eurodac in the European interoperability framework

-DubliNet is quite old, therefore a more modern, user-friendly communication system can be developed

-Increase the piloting, testing and innovation activities around DubliNet and Eurodac (e.g. use of other kinds of data capturing technology)

-Improve the preparation of the AG meetings (e.g. timely sharing of presentations, papers, etc.)

-Setting up and finalising the new Eurodac Regulation

-Increase eu-LISA’s role in standardisation

Threats

-The deadline for the revised Eurodac Regulation might not be kept, which could lead to delays

-There are no clear guidelines on how to deal with the use of facial images

-The connecting of the systems through interoperability components with other systems might not work sufficiently

Interoperability

Strengths

-eu-LISA’s high level architecture environment seems flexible and scalable for future enhancements

-Good support for training

-eu-LISA has the necessary competencies to support the Member States

-The work of the Interoperability AG is quite efficient

-The interoperability roadmap was very well communicated from eu-LISA to the Commission and, thereafter, approved

Weaknesses

-The planning of interface control documents (ICDs) was poor; they were not developed according to best practices for technical specifications.

-Technology decisions are not always documented and described well on the side of eu-LISA; making it hard for Member States to understand how decisions were made (not user-friendly)

-Lack of communication overall (with eu-LISA, sub-contractors)

-The necessary framework for standardisation across the IT systems is not well developed

-The timeline for the component development is too tight

-The simulators provided by eu-LISA have not proven to be as useful as expected; the Member States have started using their own simulators

-eu-LISA does not properly listen to the Member States but is rather focused on the feedback they receive from the Commission and the contractors

-The demand process is too time-consuming

-The AG is not chaired by a Member State but by eu-LISA, which Member States assess as a weakening of the interests of the Member States

-eu-LISA’s project management could be improved and be more agile

-Responsibilities are not clearly allocated (e.g. for testing activities) between the Member States and eu-LISA

Opportunities

-Development of an open-source strategy, in which between Member States could be included

-Improvement of the ICDs quality

-Establishment of pilot development of new systems by a small group of volunteering Member States, who would prepare stable environments for all Member States

-Strengthening of eu-LISA’s role from a passive system integrator to a more self-confident, proactive actor