Table of contents
1.Introduction
1.1.Purpose and scope
1.1.1.Purpose of the evaluation
1.1.2.Scope of the evaluation
2.Background to the intervention
2.1.Transition towards a proactive and evidence-based aviation safety management system
2.2.Description of the intervention logic and objectives of the initiative
2.3.Baseline and points of comparison
3.Implementation / state of Play
4.Method
4.1.Methodology and sources of information
4.2.Limitations and robustness of findings
5.Analysis and answers to the evaluation questions
5.1.EFFECTIVENESS
Establishment of mandatory and voluntary reporting systems
Occurrence collection rates
Occurrence reporting flow
Quality and completeness of data
Protection of information source and use of information for safety purposes
“Just Culture”
5.2.RELEVANCE
5.3.EFFICIENCY
5.4.COHERENCE
5.5.EU ADDED VALUE
6.Conclusions
Annex 1: Procedural information
Annex 2: Stakeholder consultation
Annex 3: Methods and analytical models
Glossary
Term or acronym
|
Meaning or definition
|
ADREP
|
Accident/Incident Data Reporting Programme
|
AMC
|
Acceptable Means of Compliance
|
ANS
|
Air Navigation Services
|
AOC
|
Air Operator Certificate
|
ATC
|
Air Traffic Control
|
ATM
|
Air Traffic Management
|
CAA
|
Civil Aviation Authority
|
CAG
|
Collaborative Analysis Group
|
CAT
|
Commercial Air Transport
|
DG JUST
|
Directorate-General for Justice and Consumers
|
DG MOVE
|
Directorate-General for Mobility and Transport
|
EASA
|
European Union Aviation Safety Agency
|
EC
|
European Commission
|
ECCAIRS
|
European Coordination Centre for Accident and Incident Reporting Systems
|
ECR
|
European Central Repository
|
ECR-ECCAIRS
|
European Central Repository for Occurrences
|
ECR-SRIS
|
European Central Repository for Safety Recommendations
|
ENCASIA
|
European Network of Civil Aviation Safety Investigation Authorities
|
EPAS
|
European Plan for Aviation Safety
|
ERCS
|
European Risk Classification Scheme
|
EU
|
European Union
|
FDM
|
Flight Data Monitoring
|
FTE
|
Full-time Equivalent
|
GDPR
|
General Data Protection Regulation
|
GM
|
Guidance Material
|
ICAO
|
International Civil Aviation Organisation
|
IFR
|
Instrument Flight Rules
|
iSTARS
|
Integrated Safety Trend Analysis and Reporting System
|
ITS
|
Interrupted Time Series
|
JRC
|
Joint Research Centre
|
MOR
|
Mandatory Occurrence Report
|
NoA
|
Network of Analysts
|
RMT
|
Rulemaking task
|
SCM
|
Standard Cost Model
|
SIA
|
Safety Investigation Authority
|
SMS
|
Safety Management System
|
SMTeB
|
Safety Management Technical Body
|
SSP
|
State Safety Programme
|
UAS
|
Unmanned Aircraft System
|
VOR
|
Voluntary Occurrence Report
|
1.Introduction
1.1.Purpose and scope
1.1.1.Purpose of the evaluation
This Commission Staff Working Document presents the ex-post evaluation of Regulation (EU) No 376/2014 on the reporting, analysis and follow-up on occurrences in civil aviation
carried out to support the preparation of the Commission’s Report on the implementation of this Regulation pursuant to Article 24 of Regulation (EU) No 376/2014. It assesses whether the main objectives of this Regulation have been achieved, in particular whether the Regulation improved aviation safety by ensuring that relevant safety information relating to civil aviation is reported, collected, stored, protected, exchanged, disseminated and analysed.
This Staff Working Document evaluates whether the Regulation contributed to reducing the number of aircraft accidents and related fatalities. It compares the initial expectations with the current situation. The evaluation highlights the improvements made in the area of occurrence reporting and analysis thanks to the Regulation and identifies some deficiencies and possible gaps. The assessment takes account of the following criteria: relevance, effectiveness, efficiency and the added value of the EU intervention. It also reviews the internal coherence of the Regulation as well as its coherence with other EU aviation safety regulations.
The main objective of this evaluation is to determine whether the Regulation has had the expected effect on safety and whether it has contributed to the reduction of accidents and related fatalities in aviation throughout the EU.
1.1.2.Scope of the evaluation
The evaluation covers the period from 2015 to 2019. All EU Member States and their stakeholders were consulted through the public consultation, online survey and cost-benefit questionnaire. Except for one Member State, Member States were represented by at least one stakeholder. Non-EU countries, including Norway and Switzerland, were also consulted, as well as other third countries not directly affected by the Regulation (i.e. Turkey, Canada, United States of America, Japan, Montenegro, and Albania).
2.Background to the intervention
2.1.Transition towards a proactive and evidence-based aviation safety management system
Regulation (EU) No 376/2014 is applicable as from 15 November 2015. It aims to prevent aircraft accidents and related fatalities through feedback and lessons learnt from occurrences. An occurrence is a safety-related event which endangers or which, if not corrected or addressed, could endanger an aircraft, its occupants or any other person. The Regulation seeks to ensure that relevant safety information relating to civil aviation is reported, collected, stored, protected, exchanged, disseminated and analysed. The industry and the competent authorities have to take safety action in a timely manner based on the analysis of the information collected. To ensure the continued availability of safety information, the Regulation provides rules on confidentiality of the reporting source and on the appropriate use of information. To this end, the Regulation grants, under certain conditions, the protection to the persons reporting the occurrences and other persons mentioned in occurrence reports and ensures that the information is used for safety improvement purposes. Finally, the Regulation ensures that aviation safety hazards are identified and appropriate risk-mitigation actions are taken at industry, national and Union levels.
This proactive and evidence-based approach is implemented by the relevant aviation safety authorities of Member States, by organisations as part of their safety management system and by the European Union Aviation Safety Agency (EASA).
Regulation (EU) No 376/2014 is complemented by an implementing regulationestablishing a list of occurrences in civil aviation to be mandatorily reported.
Before the establishment of the occurrence reporting system in Europe, the advancements in aviation safety relied mainly on lessons learned from the investigation of aircraft accidents. However, whilst the ability to learn lessons from accident investigation is a crucial element in ensuring aviation safety and preventing the accidents from reoccurring, such reactive approach had shown its limits in bringing safety improvements, notably in the context of continuous air traffic growth (which in 2014 was expected to almost double by 2030). On the other hand, the experience had shown that accidents are often preceded by safety-related incidents and deficiencies revealing the existence of safety hazards. Safety information is therefore an important resource for the detection of potential safety hazards. Consequently, the reactive system of accident investigation should be complemented by a proactive system, which uses other types of safety information to achieve effective improvements in aviation safety.
At the European level, the transition towards a more proactive aviation safety management system started with the adoption of Directive 2003/42/EC
which required each Member State to set up a mandatory occurrence reporting system (MORS). Member States had to collect, store, protect and disseminate between themselves information on certain civil aviation incidents. The Directive, however, did not include provisions related to the analysis of the data collected, for the benefit of aviation safety. Considering other reporting requirements existing in parallel, both at the European and international level, the adoption of Directive 2003/42/EC also led the Member States to adopt rather divergent approaches. These diverging approaches resulted, for instance, in the existence of multiple occurrence databases at the European level.
The clarification of reporting obligations had to be addressed in order to improve the accessibility, the quality, the analysis and the exchange of occurrence-related information. Additionally, the Impact Assessment for the proposal of a new Regulation on occurrence reporting
envisaged that the future legislative should address the growing importance of a “Just Culture”
, encouraging reporting of essential safety-related information.
2.2.Description of the intervention logic and objectives of the initiative
The 2012 Impact Assessment for the proposal for the occurrence reporting regulation identified the following four main problem drivers related to the previous legal framework established by Directive 2003/42/EC:
·The collection of occurrences was not optimal
At the time, the Impact Assessment concluded that the optimal collection of the safety-related information had not been achieved due to the following reasons:
oThe scope of reporting differed between the Member States, which resulted in discrepancies in Member States reporting levels
oIndividual reporters were afraid of repercussions for submitting occurrence reports
oNo obligation to establish voluntary reporting schemes was laid down in the Directive
oThere was an insufficient clarity in reporting obligations and in the flow of information
Consequently, the specific objective of the Regulation is to ensure that all occurrences which endanger or would endanger aviation safety are collected and are providing a complete and clear picture of safety risks in the EU and its Member States. To achieve this objective, following operational objectives were set:
oEstablishment of the mandatory reporting systems
oEstablishment of the voluntary reporting systems
oReaching a higher occurrence collection rate through harmonisation and classification of reporting obligations
oClarifying and development of the legislative requirements on “Just Culture”
oClarifying the flow of information
To achieve these operational objectives, the Regulation sets out the specific provisions on the mandatory and voluntary reporting, establishes a regime on the protection of information source and sets the related requirements for the Member States to ensure the protection of reporters and it provides for the specific rules facilitating the exchange of safety information between the stakeholders.
·Suboptimal data integration, low quality and incomplete data
High quality safety-related data is crucial for identifying and assessing safety hazards posed by occurrences. The Impact Assessment identified low data quality and incompleteness of data provided in the occurrence reports as two main causes of the data integration, needed for comprehensive understanding of potential safety deficiencies, being suboptimal. To remedy the situation, the specific objective was set for the Regulation to make sure that occurrence reports stored in the national databases and in the European Central Repository (ECR) are complete and contain high quality data.
Establishment and management of the ECR, standardised data entry processes, establishment of mandatory data fields and establishment of quality checking processes are the operational objectives leading to the achievement of this specific objective.
·Legal and organisational obstacles for ensuring adequate access to information contained in the ECR
The regime established by the Directive on the access to the data contained in the ECR led to a situation where due to the requirement to de-identify the data contained therein, Member State authorities and EASA did not have an access to crucial occurrence data. At the same time, Member States would also refrain from uploading the data into the ECR due to fears that such data could be used by other stakeholders for other purposes than safety (e.g. benchmarking).
Consequently, the specific objective of the Regulation is to ensure that all safety-critical information stored in the ECR is accessed adequately by the competent authorities and that they are used strictly for safety-improvement purposes. Hence, at the operational level, the Regulation provides for the granting of access rights to the ECR information to appropriate safety authorities and establishes rules on confidentiality when data is transferred to the interested parties and safeguards against misuse of data.
The Regulation therefore contains the detailed rules on the access to the ECR, granting the full online access to the ECR to any entity entrusted with regulation of civil aviation safety or any safety investigation authority within the Union and further specifying the conditions under which the interested parties listed in Annex II to the Regulation can request the information contained in the ECR. The Regulation lays down the requirements on confidentially and protection of data gathered via the occurrence reporting systems.
·Lack of occurrence analysis at Member State level and at European level and of appropriate corrective and preventive actions
Even where the data was collected under the previous legal framework, the Directive did not specify how the information collected should be used in order to contribute to safety improvements. Therefore, the Regulation aims at ensuring that reported occurrences are effectively analysed, that safety hazards are identified and addressed, where relevant, and that the safety effectiveness of actions taken is monitored.
To achieve this objective, following operational objectives were set:
oTo analyse occurrence data and to identify actual or potential safety hazards
oTo adopt preventive or corrective actions where appropriate
oTo oversee the effectiveness of the actions taken
oTo create a common EU risk classification scheme for classifying occurrences
To this end, the Regulation contains the provisions on the occurrence analysis and follow up at both Member States and EU levels. The Regulation also provides for the development of the European risk classification scheme and confers upon the Commission the necessary delegated and implementing powers to define the scheme in the EU legislation and to set the arrangements for its implementation.
Figure 1:
Intervention logic of Regulation (EU) No 376/2014
2.3.Baseline and points of comparison
Baseline analysis was prepared as part of the Support Study to this evaluation.
The baseline analysis was built on the qualitative analysis conducted in the 2012 Impact Assessment by using quantitative methods to forecast how the selected indicators would have evolved without the Regulation (EU) No 376/2014 in place. The methodology used is the Interrupted Time Series (ITS) methodology, relying on comparison of time-series data before and after intervention, built upon the basic assumption that the outcome variable would not be altered if there were no intervention. The baseline scenario for the selected indicators was forecasted through time-series, capturing the historic data evolution of these indicators themselves and other independent variables with time, and projecting this same behaviour into the future as per the actual independent variables evolution. In the issue at hand, time-series were used to forecast selected indicators from 2015 up to 2019, based on their historic evolution (from 2009 to 2015) and the evolution of other variables. After that, the Baseline results were compared with the ones actually recorded to derive conclusions on the intervention impact.
To model the counterfactual scenario under which the Regulation had not been adopted, the study analysed a number of parameters and their evolution over a period of time. The parameters included the recorded growth in the level of reporting, the levels of reporting per type of operation, stakeholder and Member State, as well as percentage of completeness of ECR fields, the amount of civil aviation accidents and incidents, and other variables such as traffic movements or a number of safety recommendations and safety actions.
In the counterfactual scenario, due to the increasing air traffic in the EU, the occurrence reporting would have continuously increased at a modest pace for some stakeholders. However, since the entry into force of the Regulation higher number of occurrences than forecasted is reported. Such improvement in occurrence reporting is evidenced across all aviation domains which are subject to reporting requirements.
Without the Regulations’ provisions on data quality, the level of completeness of the occurrence reports in the ECR would be considerably lower. Thanks to the Regulation, occurrence reports are approximately 10% more complete.
Finally, while impossible to conclude with certainty, the increased reporting may have contributed to the overall improvement of aviation safety at the European level especially by improving safety awareness, identification of safety risks and increased safety promotion. This might have had an indirect effect on the reduction of the total number of accidents compared to the counterfactual scenario.
On a more general level, without the Regulation, the voluntary reporting systems in organisation established in the EU Member States would have not existed. The Regulation allowed also for the protection of identity of the reporters of occurrences in civil aviation.
3.Implementation / state of Play
The provisions on the establishment of mandatory and voluntary reporting, set out in Articles 4 and 5 of the Regulation, were fully implemented by both the Member States and EASA. In practice, this means that each Member State has established both mandatory and voluntary reporting mechanisms at the level of its aviation authority. Similarly, EASA has a functioning mandatory reporting system in line with the requirement set out in Article 4(4) and voluntary reports can be submitted directly to the Agency and the Member States via the European Reporting Portal.
Some gaps have been identified in the implementation of these requirements by the organisations. While some organisation were already subject to the mandatory reporting system requirement prior to the Regulation’s entry into force, some other organisations had to develop it. This resulted in some organisations reporting less than others. Similarly, in the field of voluntary reporting, some organisations indicated that they did not have a voluntary reporting system in place. Finally, some organisations do not distinguish between voluntary and mandatory reporting systems, having only one reporting system in place.
The Regulation lays down the requirements for the collection, storage and exchange of information in Articles 6 and 9. Those requirements were implemented in full by the organisations as well as by the Member States and EASA. Member States established mechanisms for the exchange of information with other Member States and with EASA, and for the transfer of such information into the ECR.
The Regulation also lays down specific requirements for the management of the ECR, standardised data entry processes and quality checking processes. Majority of those provisions were fully implemented by the organisations, Member States and EASA. Nonetheless, some organisations do not have in place a formal quality checking process as required by Article 7(3). In addition, the development of a common European risk classification scheme was implemented partially. While the Commission developed the scheme by the deadline of 15 May 2017, as envisaged in Article 7(5), it still has to define the scheme in accordance with Article 7(6). The definition of ERCS together with the arrangements for its implementation is ongoing.
Articles 10, 11 and 12 stipulate the arrangements for access to the information stored in the ECR. Those provisions are fully implemented by the Commission, which is responsible for the processing data stored in the ECR and by the Member States (so called “points of contact”) who assist the Commission in relation to the requests for information stored in the ECR.
The implementation of the Regulation’s provisions related to “just culture” and the protection and appropriate use of the source of information was closely analysed. There are few shortcomings in the implementation of those provisions. While it was found that all Member States and majority of the organisations put in place processes to ensure anonymization of occurrence reports, the issue with traceability – i.e. the possibility to infer the persons concerned in an occurrence report based on the circumstances described in the report – remain, especially in the smaller organisations with less employees.
Article 16(11) lays down an obligation for the organisations to adopt internal “just culture” rules. The available information is not conclusive on whether this requirement has been fully implemented by all organisations.
The most significant shortcoming in the implementation of the Regulation was found in relation to the obligation laid down in Article 16(12) for the Member States to designate a body responsible for the implementation of the “just culture” principles. Based on the reviews of the reports on the functioning of the “just culture bodies” submitted by the Member States in accordance with Article 16(13) and additional information received from citizens, the Commission identified 11 Member States that failed to designate a “just culture body” responsible for the implementation of paragraphs 6, 9 and 11 of Article 16. As a consequence, the Commission opened infringement cases and sent out 10 Letters of Formal Notice and one Reasoned Opinion to the non-compliant Member States. The Commission also sent out a number of EU Pilots to the Member States that stated in their reports to have designated a “just culture body” but where there were still doubts as to whether such body has the competencies to discharge the function envisaged in the Regulation.
Similarly, there has been a limited progress in the implementation of Article 15(4) of the Regulation, which aims to ensure the cooperation of the competent authorities responsible for the collection, evaluation, processing, analysis and storage of the occurrence reports and the judicial authorities in the Member States via advanced arrangements. It appears that only limited number of Member States put such arrangements in place and some of those arrangements do not comprehensively cover the details of the cooperation.
Finally, the Regulation also lays down in Article 13 rules on the analysis of occurrence reports and on the adoption of preventive or corrective actions. EASA and the organisations across the EU fully implemented that provision. The Member States do use the analysis of the occurrence reports to identify potential or actual safety hazards. However, some Member States upload reports into the ECR that are relevant to other Member States, without duly informing them. Consequently, the concerned Member States are not aware of those occurrence reports and cannot make use of the information contained therein.
When it comes to the adoption of preventive or corrective actions, those are fully implemented by EASA and Member States who take preventive actions identified through the analysis of the occurrences. At the organisations side, the situation depends on the maturity of the organisation’s occurrence reporting and analysis systems. Organisations with more mature systems are capable of identifying and acting upon potential safety hazards, other organisations, with less mature systems, identify less safety hazards and take less preventive or corrective actions.
4.Method
4.1.Methodology and sources of information
A support study was carried out by an external contractor to provide input to this evaluation. In accordance with the Terms of Reference, the study was structured along eight consecutive Tasks, as depicted in the figure below:
Figure 2: Support Study Methodology Structure
The methodology for this support study was based on:
·Desk-based research, including a range of legislative documents as well as previous studies and other relevant reports that were coded and analysed to form the basis of the study and informed the assessment of the current state of implementation;
·Stakeholder consultation via an online survey, cost-benefit questionnaires, in-depth interviews, Member State experts and industry stakeholders meetings, and an open public consultation to supplement secondary evidence with views and opinions from the wide range of stakeholders; and
·In-depth analysis of the primary and secondary data collected, including an assessment of the state of implementation, an analysis of the baseline, a cost assessment, and five analytical case studies presenting results for a range of thematic areas within civil aviation.
All findings and results were triangulated and synthesised to provide evidence-based answers to the evaluation questions.
The Study was structured according to an evaluation matrix, presented in Annex 3. This matrix operationalised the study’s evaluation questions, setting them out against the sub-questions, indicators and judgement criteria that will be used to answer them. It was developed on the basis of an in-depth understanding of the Regulation’s intervention logic, depicting what it aims to achieve and how, as presented in Section 2.2 above.
The field research consisted of a public consultation as well as a range of targeted consultation activities. Seventeen stakeholder groups were consulted in the context of the study by different means.
·An online survey was launched on 13 November 2019. The survey remained open for three weeks and formally closed for responses on 9 December 2019. It was targeted to a range of different stakeholder groups, including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. The online survey included questions on the effectiveness, relevance, and EU added value of Regulation (EU) No 376/2014, as well as few questions pertaining to the degree to which the Regulation has been implemented in the Member States. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it (i.e. started completing the survey but did not answer all questions). Considering the efforts made to reach as broad a respondent group as possible, including through reminders/follow-ups, this is considered a decent number of respondents.
·Targeted interviews aimed to complement the questionnaires by collecting more details on why stakeholders hold certain opinions, as well as gather additional evidence in relation to evaluation questions for which qualitative data was judged to be an important source. A total of 40 interviews were planned out of which 39 were effectively carried out with a sample of key stakeholders per main stakeholder type.
·In terms of expert meetings, three workshops were organised, two with competent authorities (September and October 2019 in Cologne, Germany) and one with the aviation industry representatives (November 2019 in Brussels, Belgium), with the aim to gather data to feed into the answers to the evaluation questions.
·Finally, the Commission launched an open public consultation on 7 November 2019 to gather the opinions on the Regulation from the broader stakeholder community, specifically relating to the relevance, effectiveness, coherence, efficiency and EU added value of the Regulation, and on specific questions related to the concept of “just culture”. The public consultation was open for responses from the general public until 30 January 2020. A total of 198 responses were received,
Furthermore, in order to complement the study with factual information, desk research on several related issues was carried out. The aim of the desk research was to collect, organise and analyse relevant information from relevant secondary sources. This included statistical data, legislative documents and their related implementing acts, and relevant reports and studies, with the intention of collecting both quantitative and qualitative evidence to complement the primary evidence collected through field research.
·EU Legislation
·ECCAIRS cases
·EASA documentation
·Existing studies and literature.
·Five thematic case studies, namely Air traffic control, commercial air transport (CAT) operations of large aeroplanes and related design and production, general aviation manufacturer of Non-Complex aeroplanes, Unmanned Aircraft Systems (UAS) with a Maximum Take-Off Mass below 150 kg and ground handling.
The proposed sample of case studies was well balanced in order to cover instances of occurrence reporting, analysis of the reported occurrences, its quality as well as the follow-up actions taken by the relevant stakeholders. The case studies illustrate the instances of both voluntary and mandatory occurrence reporting and were selected to also provide insights into exchange and dissemination of information, confidentiality, and protection of the information source. The objective of the case studies was to take findings from stakeholder consultations (including surveys and interviews) and provide an insight on the impacts of the Regulation in specific operational scenarios.
The data collected was used to respond to the evaluation questions. All the analytical findings constitute the basis for the assessment on how the Regulation has scored on the evaluation criteria. Each of these criteria was addressed through evaluation questions, as presented in the evaluation question matrix presented in Annex 3.
In the Commission, an Inter-Service Steering Group provided advice and monitored the progress of the exercise. Being composed of members from different Commission services and having the necessary mix of knowledge and experience, the Steering Group brought together a range of different perspectives and provided the necessary input, in particular where the evaluation touched different policy areas.
4.2.Limitations and robustness of findings
Even though the evaluation was designed to ensure the robustness of the data supporting the findings presented in this Staff Working Document, some limitations to the robustness of certain data were identified while drafting the support study.
First, the responses from the online survey came mostly from the competent authorities in the Member States (34 out of 99), which biased the results of the survey towards that stakeholder group. To mitigate this bias, the results of the stakeholder groups were analysed separately, where relevant.
Secondly, there was a limited response rate to the cost-benefit questionnaires, especially from the industry stakeholders. This made the assessment of costs incurred by the industry brought by the Regulation more difficult.
Thirdly, the contractor initially encountered some difficulties in scheduling the targeted interviews with the selected representatives from various stakeholder groups, in essence due to a lack of availability by the persons and/or the entities targeted for such interviews. The impact of such situation on the data gathered was however mitigated by extending the period in which the interviews were scheduled.
5.Analysis and answers to the evaluation questions
This section provides the analysis and the results for the five evaluation criteria of effectiveness, efficiency, relevance, coherence, and EU added value.
The findings presented are based on the results from desk research, as well as results obtained through stakeholder consultations.
5.1.Effectiveness
This subsection assesses whether the Regulation has been effective in achieving the intended objectives and in particular the improvement of aviation safety in Europe. It in particular assesses how the specific objectives and the operational objectives of the intervention have been implemented.
Figure 3: Steps in the occurrence reporting, analysis and follow-up process
The first specific objective of Regulation (EU) No 376/2014 is to ensure that all occurrences that endanger or would endanger aviation safety are collected and are providing a complete and clear picture of safety risks in the EU and in the Member States.
Establishment of mandatory and voluntary reporting systems
Requirements for mandatory occurrence reporting already existed prior to the adoption of Regulation (EU) No 376/2014 and they did not therefore need to be re-established. The Regulation has however filled gaps by developing the common EU standards for reporting, across different industry domains and Member States. As such, the Regulation’s requirements are broader in terms of scope than of its predecessor Directive 2003/42/EC.
Mandatory and voluntary occurrence reporting systems are indeed in place at the Member State level, at EASA level and in the majority of concerned organisations. In addition, the European Reporting Portal was established by the Commission services.
The majority of Member States do not distinguish between the two types of reports in their respective databases, as they rather prefer encouraging organisations to submit all reports to them, regardless of their status. As a consequence, it is difficult to assess the number of voluntary reports submitted, as there is no easy way to distinguish between MOR and VOR stored in the ECR.
The Study concluded that there are no major concerns in terms of coverage of who has to report. Moreover, EASA noted that the Regulation omits ‘declared organisations’.
A related problem for EASA is that it also oversees organisations established in third countries, which are subject to reporting requirements under Regulation 2018/1139 but not Regulation 376/2014, which raises questions in terms of a level playing field and what creates a burden on EASA who has to define ‘equivalent’ occurrence reporting, analysis and follow-up requirements for those organisations.
Occurrence collection rates
An expected outcome from the effective implementation of Regulation (EU) No 376/2014 is the increased number of reports submitted and thus a greater amount of aviation safety information. One of the key objectives of Regulation (EU) No 376/2014 is to increase the available data on occurrences. This objective has been met. The total number of occurrence reports stored in the ECR has increased significantly since the entry into force of the Regulation. In 2015, there were just over 200,000 occurrence reports in the ECR in the given year. As shown in the Figure 4 below, in 2019, the total number of reports transferred to the ECR in one year reached 291,458. Cumulatively, the number of occurrence reports contained in the ECR more than doubled during this period: from 1,125,264 reports at the end of 2014 to 2,548,578 reports in 2019.
Figure 4: Number of records in the ECR (source: JRC overview of ECR-ECCAIRS content)
As shown in the Figure 5 below, there would have been an increase in the amount of collected occurrences even in the counterfactual scenario, without the Regulation in place. This would have been directly correlated with the annual traffic increase. However, since entry into force of the Regulation, the collection rate has been increased significantly compared to the expected trends. This therefore suggests that the Regulation facilitated the increase in the occurrence reporting across the EU.
Figure 5: Number of occurrences reports issued in the evaluation baseline at European level
However, according to the views expressed by stakeholders in the online survey, no conclusion can be made on whether all relevant occurrences are reported and whether the reported occurrences provide a complete and clear picture of safety risks across the EU. This can in part be attributed to the difficulty to measure whether some occurrences pose an immediate, significant safety risk and hence whether they should be reported.
There is a divergence in the views expressed by the Member States and the industry stakeholders. The authorities in the Member States believe that there has been a clear improvement in understanding safety risks through reporting. Although they acknowledge that it is likely that not all occurrences are reported, they have seen a clear improvement, not just in numbers but also in the quality of the occurrence reports. They believe that the picture of safety risks is adequate. Conversely, the stakeholders representing organisations or associations stated that not all occurrences are reported and hence the general understanding of safety risk might not be fully adequate.
Occurrence reporting flow
The reporting flow (i.e. who reports to whom and when) is well understood by all stakeholders involved, although some issues have been identified. Sometimes the details of occurrences are uploaded to the ECR by one Member State with no notification to the Member States concerned. However, there is a general expectation that the new ECCAIRS 2 software will help on this aspect.
Furthermore, the industry stakeholders complained about the lack of transparency on the processing of occurrence reports after they have been submitted to the competent authority and stored in the ECR. 23 out of 25 industry representatives in the workshop expressed their view that it is not very clear or not clear at all what happens with the occurrence reports once they are submitted to the competent authorities. The industry also expressed disappointment on the lack of a subsequent feedback from the authorities.
Furthermore, some stakeholders also expressed the view that the reporting timelines provided in the Regulation are too tight. The 72-hour time limit set out in the Regulation seems in particular to be an issue for the air navigation service providers and commercial air transport operators. However, it should be noted that in exceptional circumstances the Regulation gives possibility to submit the occurrence reports after the 72-hour time limit.
The second specific objective of Regulation (EU) No 376/2014 is to ensure that the occurrence reports stored in the national databases and the ECR are complete and contain high quality data.
Quality and completeness of data
Regulation (EU) No 376/2014 underlines the importance of high-quality and complete data. Overall, there has been a considerable improvement in the quality and completeness of the occurrence data contained in the ECR as well as in the national databases. This has been achieved thanks to the key requirements mandated in the Regulation such as a standardised data entry process, minimum information to be included and quality checking processes.
An important EU contribution is the ECR
that stores all occurrence reports collected in the EU. Member States and EASA are required to transfer to the ECR all information relating to safety stored in their respective databases in a timely manner. The national data is integrated in the ECR and checked in accordance with quality criteria.
The ECR consists of two physical databases – the European Central Repository for Occurrences (ECR-ECCAIRS) and the European Central Repository for Safety Recommendations (ECR-SRIS). Access to the ECR is subject to strict rules and mainly limited to the European competent authorities.
Throughout the period under review, the European Commission’s Joint Research Centre (JRC) provided the operational services related to the ECR-ECCAIRS. As identified by the JRC, the ECR-ECCAIRS is fed with data originating from any of the 31 contributing States as well as from EASA on a scheduled basis, with a maximum timespan of 1 month.
Following a change of strategy and after an impact study, the Commission services decided to replace the current ECCAIRS platform by a fully re-engineered platform called ECCAIRS 2, and to transfer the management of the ECCAIRS system from the JRC to EASA as of 2020.
ECCAIRS 2 was released in October 2020 and it is expected to solve many of the issues reported by competent authorities in terms of user-friendliness, system performance and training. It is a web/cloud-based solution with a user-friendly interface. It will integrate the local national repositories (without duplication/copying) into a regional view and integrate into the Data4Safety (D4S) programme of the Commission/EASA, subject to the strict rules on protection of information contained in the ECR imposed by Regulation (EU) No 376/2014 and applicable data protection rules. ECCAIRS 2 will allow the competent authorities to exploit advanced analytics, insights and auto-improved data quality. ECCAIRS 2 is likely to have a further positive impact on the effectiveness of Regulation 376/2014 as a whole.
To furthermore ensure the operational objective as regards the completeness of the data contained in the occurrence reports, Annex I to the Regulation sets out a list of mandatory data fields to be filled when uploading an occurrence report. The list of mandatory data fields can be updated by means of delegated acts, thus giving the Commission the flexibility to adapt those data fields, when appropriate and to reflect developments that may occur in the aviation sector. There are diverging opinions about that list. Some stakeholders think that not all mandatory data fields are relevant for all the reporters or reporting organisations. Furthermore, both organisations and Member States noted that the requirement of completing the mandatory data fields has produced an additional workload for both reporters and authorities. This is not the view of EASA, as it considers that all eight (8) common mandatory data fields are relevant, however those should be completed only if applicable.
Quality checking procedures should be implemented for checking the data quality of the occurrence reports thus avoiding inconsistencies and completing missing parts. Based on the collected evidence, it can be concluded that quality checking processes have at least partially been implemented by organisations and competent authorities, and they are implemented in the ECR with feedback being sent on the success/failure to the submitting body. It cannot be concluded with certainty that this was as a result of the Regulation as these processes already existed. In terms of resources, there are strong concerns in the Member States about the resources needed to do this task, as it is performed mostly manually. The use of digital tools (e.g. ECCAIRS 2, D4S) would help.
The third specific objective of Regulation (EU) No 376/2014 addresses the competent authorities’ adequate access to the safety-critical information contained in the occurrence reports, and the strict use of such information for safety-related purposes, only.
Protection of information source and use of information for safety purposes
Regulation (EU) No 376/2014 mandates for all organisations and competent authorities in the Member States to take the necessary measures to ensure the appropriate confidentiality of the details of occurrences obtained by them (i.e. mandatory and voluntary reports, as well as the data taken from the ECR) and to process this data in accordance with applicable European rules on data protection. It requires that competent authorities in the Member States as well as EASA, do not record any personal details in their respective databases to prevent a transfer of personal data to the ECR.
Based on the evidence collected for this evaluation, it can be concluded that organisations and all competent authorities in the Member States have internal processes in place to ensure that occurrence reports are anonymised before they are transferred to interested parties, notwithstanding the high workload this creates. Despite these efforts, mistakes happen and occasionally personal data is transferred due to a human error when inputting the occurrence data into the ECR.
With regard to the operational objective that the information contained in the ECR is used solely for the improvement of the safety situation in aviation and not for apportioning blame and liability, the Regulation sets out a possibility for the competent aviation authorities to conclude advance administrative arrangements with their counterpart judicial authorities setting out the terms of using the occurrence data in judicial proceedings while seeking “to ensure the correct balance between the need for proper administration of justice, on the one hand, and the necessary continued availability of safety information, on the other”. However, almost no Member State has been able to conclude such advance arrangements with their respective judicial authorities. In fact, only seven Member States have some type of arrangement in place. Hindering factors are the lack of guidance as to the content or format of such an arrangement or lack of resources and the fact that those arrangements are only facultative.
Despite the lack of advance administrative arrangements, the information available suggests that so far the data contained in the ECR is used strictly for safety improvement purposes only. This is one of the objectives to which the Regulation has contributed the most.
“Just Culture”
Regulation (EU) No 376/2014 is the first legal act to incorporate “just culture” provisions, thus ensuring its application across all the domains of the aviation at industry as well as at the competent authority level. The Regulation requires organisations to establish specific rules and procedures for the application of the Just Culture principles to the reporting of occurrences, thus ensuring that the reporters will not be subject to any form of punishment or prejudice for the occurrences reported, unless their conduct amounts to gross negligence or wilful misconduct. This should motivate the reporting at every level of the organisation.
From the evidence collected, there are indications that such rules are being formally established by the concerned organisations, however the evidence is inconclusive as regards the actual compliance with those rules by the organisations. Some representatives interviewed during the stakeholder workshop of November 2019 spoke of few instances, which appeared to be in direct contradiction to the “just culture” principles.
At the national level, the Member States are also required to put in place measures ensuring the protection of reporters and ensuring that the reporters will not be prosecuted in relation to the information shared in the occurrence report, unless their conduct amounts to gross negligence or wilful misconduct. To ensure such protection, the Member States are required to designate a body specifically responsible for the implementation of these “just culture” principles, as contained in paragraphs 6, 9 and 11 of Article 16. However, only a limited number of Member States have complied with this requirement, which led the Commission to opening a number of infringement proceedings. It should also be added that many industry stakeholders are unaware of the existence of a “just culture” body in their State, with only 4 out of 31 participants in the industry workshop answering positively when asked about their awareness of the existence of such body.
At the industry level, the European social partners, members of the sectoral social dialogue committee for civil aviation (CANSO, ETF, ATCEUC), have jointly agreed on the “ATM Just culture tool box”. It targets staff and managers within organisations providing air traffic management / air navigation services and sets guiding principles for implementing Just Culture and fostering a healthy and open reporting culture.
Ensuring that the all reported occurrences are effectively analysed, safety hazards are identified and addressed, where relevant, and the safety effectiveness of such actions is monitored, is the fourth specific objective of the Regulation.
Analysis of occurrences and identification of actual or potential hazards
The Regulation set outs the obligation for the organisations, Member States and for EASA to develop processes to analyse the information contained in the occurrence reports in order to identify safety hazards associated with those occurrences. Furthermore, the competent authorities and EASA are mandated to exchange and analyse safety information contained in the occurrence reports at the EU level.
Whereas the Study concluded that organisations, generally speaking, do have such processes in place, it should be noted that majority of the organisations already had such processes in place before the Regulation, as part of the requirements related to safety management system (SMS) contained in the implementing rules in the area of aviation safety. Nonetheless, there are organisations (e.g. ground handling), which subsequent to the Regulation’s entry into force had to introduce the processes to analyse occurrences. However, the study noted the problems experienced by such organisations in setting up such processes as the provisions of the Regulation in this regard are rather general.
Additionally, many industry stakeholders noted that the timeline of 30 days for providing the preliminary analysis of the occurrences to the competent authority is too short. Moreover, the Regulation also sets a deadline of three months from the initial report for the organisations to submit their final analysis. The majority of industry stakeholders indicated that this deadline is too tight. The Regulation allows however to extend that deadline if this is justified.
Similar to organisations, the competent authorities in the Member States all have processes in place to analyse occurrences. However, Member States differ in their opinions as regards the usefulness of such analyses and their content. Some authorities believe that the Regulation is not sufficiently clear on this aspect.
Consequently, there are differences in the degree to which analyses are carried out and the depth of these analyses. This in turn can have an impact on the identification of safety hazards and on appropriate follow up actions being taken in a timely manner. Other reasons for those differences derive also from a lack of resources (in terms of time, budget and skills) to carry out the analytical work.
The authorities noted also the “imbalance” between the requirements for such analysis and the degree of details included in the requirements for reporting, collecting, processing and storing occurrences. This imbalance leads to a high workload associated with the collecting and processing of occurrence reports and leaves little time for the analysis and follow-up.
At the EU level, there are two streams of analysis: EASA’s analysis of occurrences reported directly to the Agency by organisations for which EASA acts as a competent authority (based on occurrences stored in EASA’s own database) and a collaborative analysis done together with Member States. Those analyses result in the publication of the EASA Annual Safety Reviews and in the development of Safety Risk Portfolios (SRPs), which are used in the publication of the European Plan for Aviation Safety (EPAS).
The EASA Annual Safety Review is produced by the Safety Intelligence and Performance Department of EASA and is based on the data contained both in EASA’s own Occurrence Database as well as data contained in the ECR. As part of its risk management process, the Agency also defines and implements appropriate actions, which are subject to a continuous monitoring.
EASA works with a range of collaborative partners through two main platforms: the Network of Analysts (NoA) and the Collaborative Analysis Groups (CAGs). The NoA is set up by Regulation 376/2014 and pulls together data from different Member States to get a more general picture of safety hazards faced by the EU as a whole. The CAGs are expert groups set up by EASA that are responsible for analysing the safety of European aviation for specific aviation domains. They review available safety information, arrange in depth safety analyses and identify emerging issues. They also monitor the safety performance of their domain and provide feedback on the effectiveness of actions taken. Both systems are complementary because they provide analysis at different levels and they contribute to the Safety Risk Portfolios and to the safety actions which may be adopted in the EPAS.
Adoption and monitoring of preventive and corrective actions
At the EU level, the analyses are carried out by EASA in collaboration with the authorities in the NoA and CAGs. The outcomes of those analyses drive the development of safety actions for the EPAS and allow the setting of safety actions and priorities, not only for EASA, but also for all the different actors in the aviation supply chain. The Regulation has therefore contributed largely to the development of analyses at the EU level.
It is difficult to conclude with certainty to what extent organisations implemented preventive or corrective actions as a result of the Regulation or whether those actions derive from the pre-existing SMS requirements. Although the analyses are not necessarily resulting in a safety action, they always contribute to the greater understanding of the potential safety risks. In addition, the analyses also contribute to further staff training, safety awareness and other safety-related purposes.
The study identified that the high workload and unclear definition of the term “analysis” has resulted in a limited number of corrective and preventive actions by Member States.
Regarding the monitoring of the actions, the study confirmed that Member States and organisations put in place such monitoring. It appears, however, that those processes are largely based on the pre-existing SMS requirements and Regulation (EU) No 376/2014, did not change them. This is because the latter does not concretely suggest how the monitoring of actions should be conducted, limiting itself to establishing the obligation to conduct such monitoring.
There is a certain lack of clarity on the side of national authorities responsible for the oversight of the Regulation. While the Regulation allows for the Member States to assign one or more competent authorities responsible for establishing a mechanism to “independently collect, evaluate, process, analyse and store details of occurrences”. However, the Regulation does not provide an explicit mandate to a competent authority nominated under the Regulation to oversee the implementation of the Regulation. Furthermore, in some Member States the competent authority under Article 6(3) is a Safety Investigation Authorities (SIA), or another independent body. Those bodies however are often not empowered to conduct oversight at the level comparable to the CAA (who have an experience in the certification, oversight and enforcement in the context of national regulations and Regulation (EU) 2018/1139).
At the EU level, part of the EPAS concerns the monitoring of the implementation of EPAS actions. The implementation of safety actions identified through the domain-specific analysis carried out through the EASA CAGs is directly monitored by the relevant group. For the actions where Member States are not competent and where EASA acts as a competent authority, feedback on the implementation is regularly provided during advisory body meetings.
It can be concluded that organisations, Member States and EASA put in place processes for the analysis and follow-up of occurrences. Whereas the Regulation introduced some changes in the areas where such requirement did not exist, it nevertheless appears that some stakeholders would welcome improvements in terms of cataloguing and monitoring the corrective and preventive actions adopted as the result of the occurrence reports’ analyses.
Reduction in the number of aircraft accidents and fatalities
The general objective of the Regulation is to contribute to the reduction in the number of aircraft accidents and fatalities through a proactive approach resulting from the specific objectives detailed in the preceding sections.
It is difficult to measure the degree to which the Regulation has contributed to an actual decrease in the number of accidents and fatalities in civil aviation, both quantitatively and qualitatively.
Figure 5 shows the evolution of key statistics from commercial aircraft accidents and serious incidents of the European operators recorded by EASA. Between 2010 and 2018 the number of accidents has decreased in average terms. The decrease is more evidenced in the rate of accidents by 1 million flights (high severity). The rate of accidents has continuously decreased since 2014, while the rate of serious incidents stabilised after a peak in 2016. That peak was not an actual increase in incidents but rather a result of technical adjustments and improvements in the classification.
Figure 6: Annual safety-related occurrences in total numbers and rate per 1M movements (Source: EASA Annual Safety Reviews (2019, 2018, 2017)
After the entry into force of the Regulation, there has been a decrease in the accident rate, from 3.5 accidents per 1 million flights and 1.3 accidents per 1 million flights post-intervention (see Figure 6).
Figure 7: Number of civil aviation accidents in the evaluation baseline (Source: results from the analysis of the baseline)
The support study concluded that the Regulation contributed to the increase of the number of occurrence reports stored in the ECR, which in turn might have had contributed to the improvement of safety awareness and identification of safety risks. Consequently, Regulation (EU) No 376/2014 has likely helped to reduce the total number of accidents. Although it is almost impossible to model the exact number of accidents and other safety related occurrences in the absence of the Regulation, it can be concluded that, to some extent, the Regulation has contributed to that reduction.
Conclusions on Effectiveness
The Regulation has made a significant step towards achieving its specific objectives. It has led to improvements in the reporting across the stakeholder groups and across the EU. There has also been an improvement in the completeness and quality of the occurrence reports. However, the associated high workload related to filling in the mandatory data fields and the lack of feedback from the authorities affect to some extent the motivation to report.
There has been an increase in the number of reports stored in the ECR, suggesting that the Regulation has been effective in meeting its objectives in the area of data integration. The complexity of the use of the previous version of the ECCAIRS software and the noted gaps in the flow of information between the Member States should be solved by the new and improved ECCAIRS 2 software.
As regards the Regulation’s objectives related to the protection of information sources and the “just culture”, the Regulation provides an adequate level of protection of the reporters. However, the Regulation has not produced yet the desired progress in terms of advance administrative arrangements being concluded with the judicial authorities and the “just culture” bodies in the Member States.
Likewise, whilst the Regulation caters for improved occurrence analysis and follow-up requirements, nevertheless no improvements in terms of analysis and follow up seem to be noted. The evaluation has not revealed any issues with the regulatory provisions themselves, but points to some difficulties with the availability of adequate resources for such analysis.
In addition, the implementation of Regulation (EU) No 376/2014 and its Implementing Regulation 2015/1108 has caused increased workload for both organisations and competent authorities (the increased requirements for quality, content and timing of the occurrence reporting).
Finally, no firm conclusions can be drawn as to whether the Regulation has succeeded in achieving its overall objective to contribute to the reduction of aircraft accidents and related fatalities. Nonetheless, the data presented above show the decrease in the amount of accidents when compared to the increasing level of traffic. Due to the improved collection and analysis of the safety data from the occurrence reports and the elaboration and implementation of relevant corrective and preventive measures, it can be concluded that the Regulation has to some extent contributed to this decrease in the accident rate, along with other safety measures put in place.
5.2.Relevance
The core objective of the Regulation is to ensure that the information relevant to occurrences which endanger or could endanger aviation safety is reported, collected, processed, stored, and analysed with a view to identifying safety hazards that can then be adequately addressed through preventive or corrective actions. This proactive and predictive approach is increasingly important as there are fewer fatal accidents in the EU than globally
making the traditional reactive approach of lessons learnt from such accidents unsuitable for the continuous maintenance of high levels of aviation safety.
The aviation sector has experienced important developments since the entry into force of the Regulation. In view of the emerging technology advancements, as well as advancements in digitalisation, the study assessed whether Regulation is relevant and fit-for-purpose, notably given the upcoming rapid proliferation of the Unmanned Aircraft Systems (UAS) and the growing importance of cybersecurity.
Unmanned Aircraft Systems
Thanks to the recent amendment to the Regulation brought by Regulation (EU) 2018/1139, reporting of occurrences and other safety-related information involving unmanned aircraft falls within its scope. However, in case of occurrences and other safety-related information involving unmanned aircraft for which a certificate or declaration is not required pursuant to the Regulation (EU) 2018/1139, the Regulation (EU) No 376/2014 only applies as long as the occurrence results in a fatal or serious injury to a person or it involves aircraft other than unmanned aircraft.
As said, the occurrences involving the UAS fall within the scope of the Regulation as amended by Regulation (EU) 2018/1139. However, given the novel nature of the UAS operations and the recent entry into force of the specific regulatory framework regulating such operations there might be a need to adjust the list of mandatory data fields in the Annex to the Regulation to reflect the specificities of UAS operations. An issue to be closely monitored once reporting of occurrences with UAS aircraft advances.
The Regulation however offers relatively simple solutions in the possibilities to update both the list of occurrences to be mandatorily reported and the Annex containing the mandatory data fields in order to make the Regulation more fit for this type of operations. It follows that the Regulation continues to be relevant in the area of UAS operations.
Cybersecurity
Cybersecurity is an important new challenge in the aviation safety sector. The growing risks emanate from the use of new technologies and new devices connected to the aircraft systems and for which there are currently no guarantees that there won’t be any interferences
. It is expected that in the context of further smart and sustainable mobility developments, new technologies will continue to develop.
Though the cyber aspect is not directly addressed in Regulation (EU) No 376/2014, nor in its Implementing Regulation 2015/1018, it is possible to address such aspects under the mandatory occurrence reporting categories of ‘meteorological conditions or security-related occurrences’
, ‘failure of ATM system security’
or ‘aerodrome security related occurrences’
.
Cybersecurity should figure as one of the Key Risk Areas in the common European Risk Classification Scheme (ERCS), currently under development. It is expected that, similarly to UAS, a possible review of the mandatory data fields in light of potential cybersecurity threats will be needed when more experience is gathered.
Relevance to other recent developments
Some EU legislation adopted after the entry into force of Regulation (EU) No 376/2014 has introduced a number of provisions that might impact Regulation (EU) No 376/2014. This concerns Regulation (EU) 2018/1139, the General Data Protection Regulation (GDPR)
, Regulation (EU) 2018/1725 on the protection of natural persons with regard to the processing of personal data by the Union institutions, bodies, offices and agencies and the Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law (the “Whistleblowing Directive”)
. In particular, the Whistleblowing Directive contains a number of provisions detailing the manner in which reporting channels should be set-up, and how breaches of the Regulation should be followed-up as well as measures of protection of reporting persons. Nevertheless, none of the above instruments affect the continuing relevance of Regulation (EU) No 376/2014.
Furthermore, no concerns have been identified in relation to the environmental developments that might impact the continued relevance of the Regulation.
Market and social developments have also been considered. The air transport sector has grown by 26% between 2014 and 2018 in terms of the number of passengers transported and its growth is expected to continue with a commensurate rise in competitiveness and commercial pressures to reduce costs. This could have an impact on the employment patterns (e.g. such as hiring aircrews via intermediaries or as self-employed or so-called "pay-to-fly"). Under such context people might feel less safe to report or sharing safety experiences being afraid that such reporting might put their jobs at risk. Therefore, the concept of “Just Culture” introduced by the Regulation is even more relevant.
Conclusions on relevance
Even in the light of the recent developments in the aviation sector, involving the rapid increase in the UAS operations and the increasing threats related to cybersecurity, the Regulation continues to be relevant and able to capture those trends. Where some adjustments to the Regulation’s provisions could be desirable, e.g. the updates to the list of mandatorily reported occurrences to better reflect the UAS occurrences or mandatory data fields to gather the UAS-specific data, the Regulation offers enough flexibility to introduce those changes in a timely and efficient manner.
The Regulation also continues to be relevant in a broader socio-economic context. With its rules on the “just culture”, it offers an adequate level of protection to the reporters to foster reporting even in the increasingly more competitive environment.
5.3.Efficiency
The global average annual financial impact of Regulation (EU) No 376/2014 for the European aviation industry as a whole has been estimated to be around €15.3 million in 2018. The apparent increase in costs due to Regulation (EU) No 376/2014 is mainly caused by a sustained increase in the workload associated with its implementation, mostly the increasing number of occurrence reports collected.
Over 2014–2020 period, the administrative burdens borne in the implementation of Regulation vary significantly across Member States, industry key players and years, ranging from €7.32 million in 2015 to more than €15 million in 2018.
The total annual cost of Regulation (EU) No 376/2014 for national competent authorities is estimated to be €7.7 million. There is, however, significant variation between Member States and years, ranging from estimated annual costs of €50,000 to more than €400,000. Over time, the total costs incurred by Member States due to Regulation (EU) No 376/2014 have increased from about €2 million in 2014 to almost €8 million in 2019, indicating a significant increase resulting from the need to process the large volume of reports. As a result, the average increase in workload is estimated at 3 full-time employees per Member State.
The annual costs incurred for the exchange of information, analysis of occurrences and dissemination are estimated to represent the largest share of total costs for competent authorities in the Member States (53%), closely followed by the collection and processing of occurrences (47%).
EASA has experienced an annual cost increase of €700,000 as a result of the implementation of the Regulation, merely in collecting and processing of occurrences (63%), analysis of occurrences, exchange of information and dissemination (16%). In addition, infrastructure costs relative to ECCAIRS updates to accommodate occurrence reporting developments have been estimated as €100,000 per year.
On the benefits side, the Regulation has introduced a benefit of approximately €66.48 million over the 2015-2019 period. The model employed, taking account of all monetised costs and benefits for the 2014-2019, results in a cost-benefit ratio of 1.14, indicating a positive return on investment.
Conclusions on efficiency
Calculations made in the context of the Support Study show that the benefits derived from the Regulation outweigh the implementation costs incurred by the various stakeholders. It follows that the Regulation has been efficient in bringing considerably more benefits than costs.
5.4.Coherence
The evaluation did not identify any contradiction, overlap, duplication or inconsistency relative to the regulatory provisions set out in Regulation (EU) No 376/2014. The various components of the Regulation complement each other to achieve its objectives.
Similarly, no major issues were identified in terms of coherence with its Implementing Regulation 2015/1018.
In terms of external coherence, the Regulation has been assessed against the following EU Regulations:
-Regulation (EU) 996/2010
No major inconsistencies or a lack of coherence has been identified. However, safety investigation authorities raised concerns that due to the lack of clarity between the two reporting systems serious incidents and accidents would be reported only to the competent authority under Regulation (EU) No 376/2014. Recital 2 to Regulation (EU) No 376/2014 explains that this Regulation should not interfere with the process of accident and incident investigations managed by national safety investigation authorities as defined in Regulation (EU) No 996/2010. In the event of an accident or a serious incident, notification of the occurrence is also subject to Regulation (EU) No 996/2010. It seems therefore, that there is a need for better coordination between the relevant national authorities, so that the serious incidents and accidents are not reported only to the competent authority under Regulation (EU) No 376/2014.
-Regulation (EU) 2018/1139
Regulation (EU) 2018/1139 and its implementing rules establish obligations on certain organisations to establish occurrence reporting systems in the context of their safety management systems. A safety management system (SMS) aims to mitigate safety risks in a proactive way before they result in aviation accidents and incidents. For this, it is important to identify those risks and control them in a systematic and regulated way.
When Regulation (EU) No 376/2014 was proposed, as reflected in its fourth recital,
a need to avoid duplication of reporting efforts was highlighted. The Impact Assessment for Regulation (EU) No 376/2014 identified however a need for a specific regulation that would cover all occurrence reporting requirements, while at the same time establishing firm rules protecting the reporters and fostering the ‘just culture. At the time Regulation (EU) No 376/2014 was adopted it covered areas for which EASA had no parallel competence. That situation has changed over time with the extension of EASA’s areas of activities, although it has not yet produced any parallel coverage.
The large majority stakeholders considered that both Regulations are consistent and coherent, and no specific problems were highlighted.
EASA highlighted some practical difficulties. Although there are no legal constraints, it is often very difficult to maintain and update a coherent and effective set of safety management and oversight systems when (1) the personal and material scope of those legal acts vary, and (2) the related occurrence reporting provisions are spread across various legal acts, thus rendering their understanding difficult and hence possibly hampering effective safety management.
Although the majority of stakeholders did not raise any issue in terms of coherence, consistency or duplication resulting from the co-existence of occurrence reporting provisions in those acts, a compilation of the existing Regulations into one act could be contemplated. It has to be born in mind however that certain fundamental aspects dealt with by Regulation (EU) No 376/2014 do not fall under EASA’s remit.
Whereas the coexistence of various regulatory provisions dealing with occurrence reporting in different EU legislative acts has not been identified as a major hurdle or an impediment in the evaluation, a consolidation of those existing regulatory provisions by creating a one-stop-for all system which still guarantees protection of reports and a Just Culture provisions could be considered.
-Other relevant EU legislation
As regards coherence of Regulation (EU) No 376/2014 with other legislation, the assessment included, primarily, a range of legislative acts in the field of aviation security
, where no inconsistencies, duplications or concerns were identified.
As regards employment and social matters, no relevant EU legislation has been identified that would require coherency assessment.
As regards protection of personal data, the coherence review concluded that there are potential inconsistencies between the General Data Protection Regulation and the data protection rules specific to EU institutions and agencies contained in Regulation (EU) 2018/1725 on the one hand and Regulation (EU) No 376/2014 on the other hand. The relevant competent authorities, organisations, EASA and the Commission while storing data on the occurrences must comply with the respective requirements laid down in Regulations (EU) 2016/679 and (EU) 2018/1725. This means, that if a report includes personal data, a relevant consent for the processing of such data as required by the GDPR must be obtained from all persons identified in the report.
However, when it comes to removal of personal data, the Regulation distinguishes between two forms of such removal. On the one hand, when uploading occurrence reports into the national databases and subsequently transferring them into the ECR, or when disseminating the information within the organisation concerned, the Regulation requires the disidentification of the data. Disidentified information is defined as “information arising from occurrence reports from which all personal data such as names or addresses of natural persons have been removed”. However, such information may still contain other contextual information, which may lead to the identification of the persons involved in the reported occurrence. Such data might however be essential for the meaningful reporting, the analysis and a follow-up of occurrences. At the same time, disidentified information can still contain personal data and hence compliance with the GDPR rules must be ensured.
On the other hand, the Regulation requires a full anonymization, i.e. removal of all personal data when it comes to access to the occurrence data by the interest parties listed in Annex II to the Regulation or when the occurrence data are published by the Member States either as individual reports or in the annual safety reports.
Finally, the Regulation’s coherence with Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law was also assessed. The Directive lays down rules for common minimum standards to ensure the establishment of internal and external reporting channels and the adequate protection of reporting persons (“whistleblowers”). The Directive explicitly states that it shall complement the existing rules on the reporting of occurrences under Regulation (EU) No 376/2014, which at the same time constitute breaches of European law. Consequently, once transposed in 2021, the Directive is anticipated to have some impacts on the regime established by the Regulation.
The Directive establishes, amongst other provisions, specific details on how reporting channels should be set-up, how reports should be followed-up, including an obligation to provide feedback to the reporting person. This is expected to complement the Regulation’s regime and potentially partly solve some of the shortcomings related to the lack of feedback from the competent authorities as identified in the section on effectiveness above.
The Directive also provides for a number of enhanced measures that the Member States shall introduce to protect the reporters and penalties for breaching the confidentiality of the reports as well as for a retaliation against the reporter, which could, in some aspects, complement the Regulation’s own regime on the protection of reporters.
In conclusion, the Regulation is both coherent with and complemented by the Directive.
Conclusions on coherence
The Regulation is internally coherent and contains no overlaps, contradictions or inconsistencies. Overall, the Regulation is coherent with other European legislation on aviation safety. The Regulation’s reporting requirements however partially overlap with the requirements contained in Regulation 2018/1139 and its implementing rules. Further analysis on the impacts of removing those overlaps, possibly through a consolidation process, could be considered.
The Regulation is also coherent with other relevant EU legislation.
5.5.EU Added Value
The Regulation has been found to bring EU added value in the following areas:
First, the Regulation has standardised the approach to occurrence reporting and analysis throughout the EU, unifying criteria and leading to increased rates of reporting and higher quality reports. Results from the baseline analysis show a clear improvement in the reporting rates and completeness of reports. The Regulation has created a common framework at the EU level, which has facilitated the harmonisation of occurrence reporting, analysis and follow-up actions across Member States and across all industry domains.
Second, the EU involvement has enabled the pooling of information and experience at the European level. This improves coordination among Member States, EASA and the Commission, for example via the Network of Analysts. All Member States are involved in the proceedings and have full access to the ECR data. It is expected that the exchange of safety information will improve with the help of modern and efficient e-tools (e.g. ECCAIRS 2, Data4Safety). The Regulation has enabled an EU-wide understanding on aviation safety that could not have been achieved by the Member States themselves and has in turn contributed to the further maturation of the EPAS.
Third, the Regulation has supported the development of the “Just Culture” and helped to increase the awareness of the importance of occurrence reporting for safety, which in turn led to an increase in the number of occurrence reports submitted.
Finally, with the upcoming replacement of the current ECCAIRS platform with a more modern and efficient software platform developed and managed by EASA and the adoption of a common risk classification scheme as required by Regulation (EU) No 376/2014 further improvements in the collection and analysis of the occurrences could be expected.
On the basis of all those elements it can be concluded that the EU added value of Regulation (EU) No 376/2014 is very high.
6.Conclusions
The Regulation has met all evaluation criteria. As the Regulation is a part of a wider aviation safety framework, it is difficult to isolate its effects. Nonetheless, the evaluation suggests that the Regulation has contributed to the overall improvement of aviation safety in the EU.
While it is impossible to draw conclusions on the exact impact of Regulation (EU) No 376/2014 on the reduction of accidents and fatalities in civil aviation, there are some indications that the Regulation played a positive role in decreasing aviation accidents in Europe.
The Regulation has made a significant step towards achieving its specific objectives. It has been particularly successful in increasing the number of occurrences reported and collected and it had a positive impact on the level of reporting. Nevertheless, some shortcomings relating to the big amount of mandatory data fields or too short reporting deadlines were identified, which could have led to an increased workload both at organisational and Member State levels and might have had an impact on the quality and completeness of the reports.
The Regulation has contributed to better collection and storage of the occurrence reporting. There has been a considerable improvement in the quality and completeness of the occurrence data contained in the ECR and national databases. This however implies a considerable workload for the involved stakeholders and especially for the authorities. Digital processing offered by the recently introduced ECCAIRS 2 software should mitigate some of those difficulties.
Concerning data protection and exchange of information, as occurrence reporting uses no IT technology which would ensure full disidentification of the reports, personal details occasionally are registered in the national databases and the ECR. In this regard better enforcement of GDPR requirements is needed, and as of December 2021, the competent authorities should also ensure the confidentiality of within the obligations established by the Whistleblowing Directive. The evaluation concluded that access to the ECR is well protected and that ECR data have been used for safety purposes only.
The Regulation however only partially achieved its objectives with regard to the protection of reporters and the "just culture”. Only a fraction of Member States established advance administrative arrangements with their respective judicial authorities to manage the access to and the use of occurrence reports contained in their national databases. Moreover, a number of Member States failed to designate a “just culture body” with the necessary competences to implement its duties and guarantees that reporters are not punished for reporting. To address this problem the Commission opened several infringement proceedings against Member States that are lagging behind.
At the EU level, the Regulation has facilitated the analysis of occurrence reports through the standardisation of requirements what resulted in the increase of quality of reported data. The annual publication of the EPAS, including key risk areas and actions to mitigate safety concerns is a good example. At a national level, however, the evaluation has identified an apparent lack of analysis and follow up capabilities. A lack of expertise, both in terms of adequate resources and procedures seem to be the root causes. Furthermore, industry stakeholders have highlighted the lack of feedback from the competent authorities on the analysis of the reported occurrences.
The Regulation’s main deficiency is the lack of clarity on the competent authority responsible for the oversight of the entire Regulation. The Regulation allows the Member States to nominate up to three ‘competent authorities’ to run the process from collection to analysis, with no further obligations on oversight. Some of those authorities seem not to be properly equipped to oversee the follow up actions.
In terms of relevance, the evaluation shows that Regulation (EU) No 376/2014 and its objectives remain relevant also in face of the latest developments in aviation, allowing for taking account of the upcoming increase of UAS traffic.
Regulation (EU) No 376/2014 has resulted in costs for both competent authorities and organisations, but the identified benefits clearly outweigh the costs incurred.
In terms of coherence, Regulation (EU) No 376/2014 is fully coherent internally and overall, it is consistent with other EU instruments, notably with Regulation (EU) 2018/1139, Regulation (EU) No 996/2010 on accident and incident investigation, Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, Regulation (EU) on the protection of natural persons with regard to the processing of personal data by the Union institutions, bodies, offices and agencies and on the free movement of such data and the Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law. The interaction between Regulation (EU) 2018/1139 and Regulation (EU) No 996/2010 warrant further consideration whether reporting and analysis obligations should be streamlined into one set of obligations.
Finally, based on the evidence analysed, it can be concluded that the Regulation brought a clear added value in the area of occurrence reporting, analysis and follow-up. It has contributed to a clear improvement in reporting rates, completeness of reports, amount of safety-relevant data available across the EU as well as to the protection of reporter and introduction of “just culture” principles.
Regulation (EU) No 376/2014 has fostered the coordination amongst Member States, EASA and the Commission via different groups (e.g. Network of Analysts). Exchange of safety information is possible via the access to the European Central Repository, set up by the EU by all Member States’ authorities (including safety investigation authorities).
Annex 1: Procedural information
1.Lead DG, Decide Planning/CWP references
Lead DG:
DG MOVE
Planning reference:
PLAN/2019/5269
2.Organisation and timing
Article 24 of Regulation (EU) No 376/2014 establishes the legal obligation to submit an evaluation report on the implementation of this Regulation to the Parliament and to the Council.
The evaluation was launched in April 2019 by establishing an inter-service steering group (ISSG). The first meeting of the ISSG took place on 12 April 2019 and discussed the detailed planning of the evaluation exercise as well as the content of a draft roadmap outlining the procedural and methodological steps to be taken. The roadmap was published for feedback for a period of four weeks. The roadmap was finalised on 22 May 2019.
The Commission launched a call for tenders for a support study on “Ex-post evaluation of Regulation (EU) No 376/2014 on the reporting, analysis and follow-up of occurrences in civil aviation”. A specific contract No MOVE/E4/SER/2019-422/SI2.811176 was signed with an external consultant Ramboll Management Consulting A/S on 23 May 2018 under the framework contract No MOVE/A3/2017-257. The external consultant delivered the first draft of the support study on 24 February 2020. The final delivery of the final report was sent on 12 May 2020 and the ISSG subsequently approved the final report and it was published by the Commission.
The evaluation study is based on the assessment of the effectiveness, efficiency, coherence, relevance and EU added value of Regulation 996/2010, and reviews the objectives of the original regulatory intervention and the performance of the intervention as compared to the initial expectations and the current situation. The study also determines whether there are overlaps with other safety-related regulatory tools.
Based on the answers received in the OPC and based on the support study produced by the external consultant, the Commission proceeded with drafting this Commission Staff Working Document (CSWD). The ISSG was consulted on the draft CSWD on 16 and 30 September 2020.
3.Exceptions to the better regulation guidelines
No exceptions to the Better Regulation Guidelines.
4.Consultation of the RSB (if applicable)
N/A
5.Evidence, sources and quality
The evaluation of Regulation 376/2014 was based on the intervention logic of Regulation (EU) No 376/2014 and a comprehensive analytical framework comprising the evaluation questions and their respective judgement criteria, indicators and information sources.
The data collection tools used to gather the relevant information consisted of desk research, targeted online survey, targeted cost-benefit questionnaires, targeted interviews, expert meetings, public consultation and five case studies.
The documents reviewed consisted of the EU legislation, reports and other communications, ENCASIA reports, ICAO documents, advanced arrangements concluded between the SIAs and national judicial authorities, safety recommendations, documents and reports by the national investigation authorities, and other sources, including academic articles and publications.
The targeted online survey covered a representative selection of stakeholders involved in the or affected by the reporting, analysis and follow-up of occurrences in civil aviation including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it.
The targeted cost-benefit questionnaire dissemination followed the same approach as that used for the online survey. In total 17 respondents completed the targeted cost-benefit questionnaire.
Targeted interviews were conducted with a range of different stakeholder groups, and a wide geographical scope, namely with European Commission officials and representatives of the following organisations: ICAO, EUROCONTROL, EASA, competent authorities in EU Member States, Safety Investigation Authorities, just culture bodies / judicial authorities, commercial aircraft operators, manufacturers, maintenance industry, airports / ground handling operators, aviation personnel, General aviation / non-commercial operations / UAS, ANSPs, ATC service providers, research centre, and passengers. A total of 39 targeted interviews were carried out.
In the context of the expert meetings, the study team facilitated two workshops and attended a third one as an observer with support from EASA and the European Commission. The first workshop was a full day workshop with competent authorities; the second workshop was a half-day workshop with the Network of Aviation Safety Analysts (NoA); and, the third workshop was a full-day workshop with a range of industry stakeholders.
In addition, the European Commission launched an online public consultation on 7 November 2019 to gather the opinions of the broader stakeholder community on Regulation 376/2014. A total of 198 responses were received.
Five case studies were conducted on the application of the Regulation 376/2014. The case studies were selected based on the criteria of the relevance vis-à-vis the evaluation questions, falling within the scope of the Regulation, being related to Member States compliance and based on the diversity and the availability of sufficient amount of details/information.
Then, an assessment of the state of implementation of the Regulation 376/2014 was carried out based on data collected through desk-based research, stakeholder consultations, data collection tools, the online survey and interviews. This information was then triangulated to enable a judgement to be made on the extent to which the requirement has been implemented.
Furthermore, an analysis of the evaluation baseline was carried out. In particular, the baseline analysis covered the effectiveness of the Regulation, particularly pertaining to the extent to which the Regulation has impacted the levels of safety in the EU. The methodology used concerns the Interrupted Time Series (ITS) methodology, which was considered the strongest quasi-experimental method to derive causal inference from the data available.
Finally, a cost-assessment was carried out. The targeted cost-benefit questionnaires formed the basis of this assessment, supplemented by estimations where needed to fill the gaps. The methodology that was applied for this assessment is the Standard Cost Model (SCM), which allows to address the costs and burdens of Regulation 376/2014 for both Member States (competent authorities) and industry stakeholders.
Annex 2: Stakeholder consultation
The objective of this synopsis is to provide an overview of the results of the stakeholder consultation carried out in order to evaluate the Regulation 376/2014 on the reporting, analysis and follow-up of occurrences in civil aviation. It includes a general analysis of the several methods used to reach the stakeholders, which were:
·An online survey launched on the 13th November 2019 and extended until the 9th December 2019;
·A cost-benefit questionnaire launched on the 13th November 2019 and extended until the 9th December 2019;
·Targeted interviews;
·Two workshops with competent authorities held on the 19th September 2019 and 30th October 2019;
·A workshop with industry stakeholders held on the 25th November 2019;
·An open public consultation launched on the 7th of November 2019 until the 30th of January 2020.
The goal was to assess to what extent the aim of the Regulation was achieved, through an independent and evidence-based investigation. It also gave the possibility to stakeholders to express themselves and share their opinion on the progress made in accident and occurrence reporting in civil aviation since the Regulation entered into force.
Online survey
Launched on the 13th November 2019, the survey was targeted to a range of different stakeholder groups, including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it. Stakeholders were often unresponsive to the launch of the survey and consequently the deadline had to be extended and reminders sent. No answers were collected from three groups: judicial or law enforcement authorities, EFTA surveillance authority, and aerodrome operators. All other groups were represented, at least by a few respondents (Figure 1). The respondents were from 28 Member States (Belgium, Croatia, Cyprus, Czechia, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland and United Kingdom).
Figure 1: Distribution of the respondents to the targeted survey over the stakeholder groups (N=99).
The online survey included questions on the effectiveness, relevance, and EU added value of Regulation 376/2014, as well as a few questions pertaining to the degree to which the Regulation has been implemented in the Member States.
Cost-benefit questionnaire
Launched on the 13th November 2019, the cost-benefit questionnaire was targeted to national authorities on one hand, and industry stakeholders on the other.
On the authority side, a total of 14 completed questionnaires were returned (Figure 2), covering 14 different Member States On the industry side, however, only 3 completed questionnaires were returned: from a ground handling service provider, an aircraft parts maintenance organisation, and an aviation manufacturer.
Figure 2: Distribution of the national authorities that responded to the cost-benefit questionnaire over the role within the framework of Regulation 376/2014 (N=14).
The cost-benefit questionnaire contained questions pertaining to the efficiency of Regulation 376/2014, including specific questions regarding the costs incurred as a result of the Regulation, and how these costs compare to the benefits achieved.
Targeted interviews
A total of 40 targeted interviews were carried out in the context of the evaluation support study. They cover a range of different stakeholder groups, and a wide geographical scope (Table 1).
Some of the interviews conducted were group interviews involving two or more stakeholders, so the total number of organisations covered is higher than the foreseen number of interviews. Issues encountered in scheduling interviews (e.g. because of the Christmas holidays, and aviation professionals having limited time to talk to us) caused delays compared to the initially foreseen timeline. However, the deadline was extended to mitigate this, and the foreseen number of interviewees was attained. Overall, the study team judges the sample consulted to be representative of the breadth of stakeholders concerned by Regulation 376/2014.
Table 1: Overview of stakeholders interviewed.
Stakeholder
|
Planned
|
Conducted
|
ICAO
|
1
|
1
|
EUROCONTROL
|
1
|
1
|
European Commission
|
2
|
2
|
EASA
|
2
|
2
|
Competent authorities in EU Member States
|
5
|
5
|
Safety Investigation Authorities (SIA)
|
3
|
3
|
Just culture bodies / Judicial authorities
|
2
|
2
|
Commercial aircraft operators
|
4
|
4
|
Manufacturers
|
2
|
2
|
Maintenance industry
|
2
|
2
|
Airports / ground handling operators
|
3
|
3
|
Aviation personnel
|
3
|
3
|
General aviation / non-commercial operations / UAS
|
3
|
3
|
ANSPs
|
3
|
3
|
ATC service providers
|
2
|
2
|
Research centre
|
1
|
1
|
Passengers
|
1
|
1
|
Total
|
40
|
40
|
The targeted interviews aimed to complement the questionnaires by collecting more details on why stakeholders hold certain opinions, as well as gather additional evidence in relation to evaluation questions for which qualitative data was judged to be an important source.
First workshop with competent authorities
A first workshop with competent authorities was held at EASA premises in Cologne on 19th September 2019, after the SMTeB meeting. Hence, it was attended by representatives of the Member States’ competent authorities, some of which are SMTeB members.
A total of 17 Member States were present at the meeting and six non-EU Member States (Moldova, Norway, Serbia, Montenegro, Iceland, and Switzerland), and EASA.
The first workshop was a full day workshop with competent authorities, which aimed to gather insights into:
1.the effectiveness, efficiency and coherence of Regulation 376/2014;
2.reporting and collecting;
3.data protection, confidentiality and just culture;
4.exchange of information, dissemination and actions.
The authorities were split up into discussion groups (ensuring a balanced geographical and gender representation in each group), where they discussed each of the above topics, following a list of predetermined questions. After two discussion sessions, participant rapporteurs were asked to present their key findings in plenary.
Second workshop with competent authorities
The second workshop with competent authorities was held on 30th October 2019 in Cologne.
It constituted of a half-day workshop with members of the network of analysts (NoA). The topics covered were therefore primarily related to analysis and follow up. Considering the limited time and space available, live audience polling software was used to gather data from participants, specifically covering questions on four broad topics:
1.storage;
2.risk classification;
3.analysis;
4.actions and dissemination.
The results of given questions were discussed by a participant panel, after which the floor was opened to additional comments from all other participants. This data was subsequently analysed in Excel, and completed with notes taken by a member of the study team who attended the workshop as an observer.
Workshop with industry stakeholders
The third workshop was a full-day workshop and was held on 25th November 2019 in the Albert Borschette conference centre in Brussels. The study team, with support from EASA and the European Commission, facilitated the workshop.
A range of different industry stakeholders were invited to attend, to get as broad a stakeholder coverage as possible (Table 2). However, it proved difficult to get some groups on board, e.g. large airlines, airports, large aircraft manufacturers, ground handling service providers, maintenance organisations. Yet, a decent balance was struck, both in terms of geographical coverage and organisations types, although many were represented by their respective associations.
Table 2: Stakeholder groups present at industry workshop.
Sector
|
Organisation
|
ANSPs
|
DFS (Germany)
|
|
ENAIRE (Spain)
|
|
PANSA (Poland)
|
ATC
|
ENAV (Italy)
|
|
FerroNATS (Spain)
|
Operators
|
A4E (Association – airlines)
|
|
EHA (Association – helicopters)
|
General aviation
|
Luxaviation Group
|
Airports
|
Brussels Airport
|
Ground handling service provider
|
ASA World (Association)
|
Maintenance organisation
|
Lufthansa Technik
|
Manufacturer
|
Dassault
|
|
ATR
|
|
GAMA Europe
|
|
ASD (Association)
|
Aviation personnel
|
ECA (Association – pilots)
|
|
EBAA (Association – operators)
|
|
IFATSEA (Association – safety electronics)
|
|
IFATCA (Association – ATC)
|
International/regional organisations
|
EUROCONTROL
|
|
IATA
|
The workshop consisted of live audience polling using the software “Poll Everywhere”, followed by comments by the predetermined participant panel, and comments from the audience. Questions covered four topics:
1.mandatory and voluntary occurrence reporting;
2.risk classification, analysis and follow-up;
3.data protection and just culture;
4.general issues related to Regulation 376/2014.
The results of given questions were discussed by a participant panel, after which the floor was opened to additional comments from all other participants. This data was subsequently analysed in Excel, and completed with notes taken by the study team.
Public Consultation
The European Commission designed and carried out a Public Consultation (PC) on the Evaluation of Regulation (EU) No 376/2014 on the reporting, analysis and follow-up of occurrences in civil aviation. The PC was launched on the 7th of November 2019 and remained open for responses until the 30th of January 2020. A total of 198 responses were received (Figure 3).
Figure 3: Contribution per stakeholder
The Public Consultaiton consisted of 21 questions and the structure of this report is as follows:
1.
Profiling questions
2.
Relevance
3.
Effectiveness
4.
Coherence
5.
Just Culture
6.
Efficiency
7.
EU Added Value
8.
Additional Comments
The answers were analysed in Excel and used to answer the evaluation questions contained in the evaluation matrix.
The questionnaire opened with a general question relating to the relevance of Regulation 376/2014. The respondents were asked for their views on whether the process of occurrence reporting is an appropriate method for collecting safety related information. Overall, the majority of respondents answered that they “strongly agree” while just a few responded “disagree”. It is worth noting that while “strongly disagree” was an option in this question, it gained zero responses (Figure 4).
Figure 4: PC Question on Relevance
With regards to the collection and analysis of occurrence information, the majority of the respondents were in agreement that the collection and analysis of occurrence reports leads to a clear picture of aviation safety risks (Figure 5). In addition, the majority agree that the occurrence reporting logic of the Regulation is effective in achieving an evidence-based aviation system. On the other hand, regarding the completeness of the data, there was not a clear majority, however, the largest support was gathered from the participants that agreed that the Regulation ensures that all safety related information which has the potential to endanger aviation safety is collected.
Figure 5: PC Question on Effectiveness
Concerning the action on safety information, the participants, although without a clear majority, mostly agreed that industry organisations or public authorities take appropriate actions based on the analysis of the occurrence received. Also, the number of respondents who agree with the opinion that the Regulation ensures that safety action can be taken in a timely manner was the highest, though without a clear majority. In addition, the majority of the respondents agree that the Regulation has improved aviation safety by ensuring that relevant safety information is reported, collected, protected, analysed and disseminated. Moreover, the majority agree with the view that annual safety reviews from Member States and EASA are both informative and provide a view of the types of aviation safety.
In relation to coherence with other legislations, the respondents were asked to highlight their agreement or disagreement with three statements that referred to the coherence between Regulation 376/2014 and other regulations and organisational rules. The statement that gathered the most support was that the Regulation was coherent with other EU related legislation. On the second statement, there was a similar finding that the majority of the respondents were in agreement that the Regulation is coherent with international aviation-related regulations. Out of the three statements, the statement that ‘the Regulation is coherent with other regulations derived from social policies’ had more of a variety of responses (Figure 6).
Figure 6: PC Question on Coherence
Regarding the encouraging reporting while retaining the need for sufficient information, there was a great variety of opinions on whether the Regulation stimulates and ensures consistency and a holistic approach to safety management. When asked to elaborate on why they disagreed with the statement, the most salient theme was that the Regulation does not act as an “efficient driver” with respect to safety management. Moreover, there was a strong proportion of the respondents that agreed with the statement that the Regulation ensures a good balance between encouraging reporting and the need to have sufficient information.
Concerning the reporting culture and protection of information, the respondents were asked if they agreed or disagreed that the Regulation is able to foster a “just culture” and positive reporting culture with respect to their national legislative frameworks. Overall, the vast majority of the respondents are in agreement that the Regulation is able to foster a just culture (Figure 7). In addition, a strong majority are in agreement that the Regulation ensures that there is enhanced protection for persons mentioned in occurrence reports. With regards to the current provisions of the Regulation in terms of ensuring that information reported is used for safety improvement purposes only, the majority of respondents answered that they agree.
Figure 7: PC Question on Just Culture
When asked about the existence of a ‘just culture’ body in Member States, the majority of the respondents were in agreement that they are aware of the existence of a just culture body in their Member State.
Two main questions were asked which directly relate to the alignment of reporting methods and other safety reporting obligations. First, the respondents were asked if the types of occurrences that are required to be reported on are well aligned with other safety reporting obligations. There is a variety of responses and no clear majority from any of the respondents with a large proportion either in disagreement or neither agree nor disagree (Figure 8). In the second question, there is a divide in opinion on where regulatory burdens have been brought about as a result of the Regulation. To explore the factors behind the decision from the people who responded “agree” or “strongly agree”, a follow-up question allowed for participants to elaborate further. An example of an element which imposes these issues are the number of mandatory fields that reporters need to fill in when completing an occurrence report. The other themes that emerged included the lack of coordination both within the Regulation and with other reporting obligations and issues regarding the lack of feedback and increased costs.
Figure 8: PC Question on Efficiency
The final section looks at the added value of the Regulation which compares the existing situation with a hypothetical scenario in which the Regulation does not exist. The analysis then identifies the changes, which can reasonably be argued to be due to the Regulation being in place, as compared to the scenario where only national measures by the EU Member States are in place. This section starts with the question if the Regulation has put provisions in place to harmonise information storage and risk classification. There is no clear majority for any of the responses. The largest response came from the respondents who agreed that the Regulation had put provisions in place (Figure 9). Then, the respondents were asked to identify if they agree or disagree that the Regulation includes a list of occurrences to be mandatorily reported by the different types of reporters. In particular, the respondents were asked to answer whether they thought this classification had improved the clarification on occurrences to be reported. The majority of responses were in agreement that there was an improvement. Moreover, the vast majority of responses think that stopping the Regulation would have a negative effect on aviation safety.
Figure 9: PC Question on EU Added Value
Results of consultation activities
|
Stakeholders have provided feedback, which has helped evaluate the Regulation, identify the issues and have also provided with some suggestions for improvement. These suggestions came mainly through the workshop, the focus groups and the open public consultation. In general, participants have considered that although there have been improvements, there are still shortcomings to be addressed.
Relevance
During the consultation activities, there was an overall agreement that the Regulation 376/2014’s objectives continue to be relevant today, as there will always be a need to reduce the number of aircraft accidents and fatalities through a proactive, preventative approach to aviation safety at EU level based on information stemming from daily occurrences. However, it was mentioned that the Regulation does not cover reporting requirements for individual UAS operators, and there is still a lack of clarity in the UAS domain in terms of what needs to be reported on.
Effectiveness
There was an overall agreement through the consultation activities that the Regulation 376/2014 has ensured that more occurrences are reported and collected than in the past, contributing to a more complete picture of aviation safety than was previously the case by ensuring that occurrence reporting is done more frequently across different domains and Member States. That being said, its impact in this regard has been greater in industry domains/Member States which did not have a mature reporting system in place prior to the inception of the Regulation.
Despite such increasing trends, the effectiveness of the reporting process has been hampered by a number of factors, notably the time and effort associated with the extensive requirements for reporting in Regulation 376/2014. This includes the long list of mandatory data fields which are not all relevant to all reporting organisations, and timelines for reporting and analysis which cannot always be met. As a result, there is a lack of voluntary reporting on safety hazards or other occurrences which are not part of the MOR system, primarily because of a lack of clarity on what constitutes “other safety related information”, and the limited resources available which instead are spent on compliance with the extensive requirements for reporting on mandatory occurrence types.
During the consultation activities, it was concluded that the Regulation 376/2014 adequately sets out what occurrences need to be reported on in principle, pursuant to Article 4(1) and Implementing Regulation 2015/1018, although it would be appropriate to conduct a thorough review with input from aviation professionals to ensure any new occurrence types are covered.
There has been a considerable improvement in the quality and completeness of the occurrence data contained in the ECR and national databases as a result of Regulation 376/2014 which put in place a standardised framework for both mandatory and voluntary reporting. However, there are still gaps in the quality and completeness of reports contained in the ECR because of different approaches to quality checking and data transfer (e.g. Member States not ‘matching’ duplicate reports), which complicates the analysis at EU level.
Other conclusion consisted on Regulation 376/2014 being partly effective in ensuring data protection. The organisations concerned by occurrence reporting and competent authorities alike have processes in place to ensure occurrence reports are disidentified so that no personal information is recorded in their (national) databases or in the ECR. However, these processes were not necessarily established as a result of the Regulation.
Many Member States have not yet designated a ‘just culture’ body. Even among those who have, only six Member States have put the necessary rules in place to ensure they are empowered to carry out the activities stipulated in Article 16(6), (9) and (11) of Regulation 376/2014.
The analysis of reported occurrences takes place at three different levels – the level of the organisations, at national level and at EU level, but it is not working optimally in each case, leading to limited identification of safety risks. Overall, there is always room for improvement, e.g. by taking into account exposure data in the analysis, rather than focusing on occurrence data alone.
Moreover, industry stakeholders are widely of the opinion that the authorities are not providing sufficient feedback on the outcome of their analysis, which makes them believe that the data they provide is not being analysed and falls into a ‘black hole’ (the ECR). Authorities are not sufficiently disseminating the results of their analysis which acts as a disincentive to reporting.
Competent authorities are required by the Regulation to oversee the implementation of actions taken by organisations for which they act as competent authority, but it is not clear how they should act in case of a lack of compliance. Beyond monitoring the implementation of actions, there is no requirement for competent authorities to oversee the correct implementation of the Regulation more broadly.
Efficiency
During the consultation activities, it was concluded that the Regulation 376/2014 has resulted in costs for both competent authorities and organisations, but based on conservative assumptions, the benefits identified outweigh the costs incurred at least to a limited extent.
Coherence
Based on the results of the consultation activities, the Regulation 376/2014 did not reveal major issues in terms of internal coherence, except for a number of inconsistencies related to the use and denomination of certain definitions in the Regulation.
The analysis of the external coherence of Regulation 376/2014 with EU legislation in the EU air transport acquis revealed a few key issues. Stakeholders criticised that the existence of different legal acts covering related requirements for the reporting, analysis and follow-up on occurrences in civil aviation leads to confusion and complexity.
EU added value
During the consultation activities, it was concluded that the Regulation 376/2014 provides clear EU added value through its contributions to effective safety management at EU level compared to what likely could have been achieved without EU intervention or at different levels of intervention. However, this added value has not been felt to the same extent across Member States and industry domains – depending on the degree to which a mature occurrence reporting culture was already in place.
Use of consultation results
|
These consultation activities served as field research in order to provide the Commission with the stakeholders view on the achievements of the Regulation 376/2014, and to determine if the aims of the regulation have been accomplished. It collected relevant data on the stakeholders opinions, which provide insight on the shortcomings of the Regulation.
It was concluded that the Regulation requirements have brought about higher rates and more quality on reporting, analysis and follow-up of occurrences in civil aviation, and a consequent improvements in aviation safety. Nonetheless, further improvement is possible. The stakeholders consultation and the OPC provided valuable feedback from the stakeholders which is helpful in finding where the shortcomings are and possible solutions for how to tackle them. The final results are a rich source of material to understand the achievements of the Regulation 376/2014.
Annex 3: Methods and analytical models
As a general principle, the methodology used during the evaluation respects the principles of objectivity, reliability and evidence based assessment, and complies with the requirements of the Better Regulation Guidelines. Where relevant, tools proposed in the Better Regulation "Toolbox" have been taken into account and made use of.
The following five criteria have been applied throughout the evaluation of Regulation (EU) No 376/2014 on the investigation and prevention of accidents and incidents in civil aviation and shall allow to determine to what extent the original objectives of this Regulation have been met: effectiveness, efficiency, relevance, coherence, and EU added value.
The following evaluation questions have been used to carry out the tasks done in the context of the evaluation and reflection each criterion used:
Effectiveness
1)To what extent have the operational objectives of the Regulation been achieved? To what extent have the specific and general objectives been met?
2)What actual observed effects (positive and negative impacts) has the Regulation had? To what extent has the Regulation contributed to these effects?
3)What were the main drivers and hindrances to the effectiveness of the Regulation? If there are significant differences between Member States, or between industry domains, please provide examples and explain in which ways and to what extent.
4)Which unexpected or unintended effects (positive or negative) have occurred as a result of the implementation of the Regulation, if any and what factors have influenced those achievements? Please specify the effects for each stakeholder.
5)Does the Regulation meet the current scope of aviation activity in EU Member States and EASA? Are all organisations and persons adequately addressed in the Regulation?
6)Does the Regulation adequately and consistently empower Member States and EASA to oversee the implementation of the Regulation in organisations?
Efficiency
7)To what extent are the regulatory costs (i.e. compliance costs, enforcement/implementation costs and administrative costs) of the Regulation proportional with the achieved its benefits? Who bears these costs?
8)Has the Regulation resulted in unnecessary regulatory burdens or inefficiencies? If so, what are the reasons for this? Is there a potential for the reduction of the regulatory costs for the main actors or for simplification?
9)Could the same results have been achieved at a lower cost? Could the use of other policy instruments or mechanisms have provided for better efficiency?
Relevance
10)To what extent are the objectives and scope which were identified at the time of adoption still adequate in the current context and how do they still correspond to the problems and needs of the aviation safety system that is currently in place? Please take into account the technological, market, political, social, environmental and legal developments such as digitalisation, big data and artificial intelligence, the growth in air traffic and its implications for air safety, the increasing complexity and dynamics of the aviation system as well as any other matters that may be relevant.
11)Which issues that arose after the adoption of the Regulation require further attention in view of the objectives pursued?
Coherence
12)Are the requirements and provisions set out in the Regulation coherent and consistent with one another? If not entirely, what are the differences, overlaps or inconsistencies?
13)To what extent is the Regulation coherent and consistent with other EU regulatory instruments under the air internal transport acquis, such as Regulation (EU) 996/2010 on the investigation and prevention of accidents and incidents in civil aviation, or the safety management and occurrence reporting requirements set out in the EASA Basic Regulation and its Implementing Rules? If not entirely, what would be the differences, overlaps or contradictions or inconsistencies?
14)To what extent is the Regulation coherent and consistent with the relevant EU legislation on safety, security, employment and social matters? Does a legal gap exist in these cases?
EU Added Value
15)What is the added value resulting from EU intervention in regulating the occurrence reporting system? Could the same results be achieved at international, national or regional level without EU intervention?
16)What would be a consequence of stopping or withdrawing the existing EU intervention?
The intervention logic present below is a visual representation of the main objectives and the causal chain of the intervention behind the Regulation 996/2010. The intervention logic links the following main elements:
General objectives Specific Objectives Operational Objectives Inputs Outputs Results Impacts
Methodology applied for cost assessment
The cost assessment focused on the costs incurred by the industry, Competent Authorities and EASA to collect, transfer, store and disseminate information on safety related occurrences. The term “costs” refers direct costs resulting from the implementation of the Regulation, including direct compliance costs and administrative burdens.
The main difference between most compliance costs and administrative burdens is that the latter only occur due to the legislative requirements of providing and collecting information while compliance costs are incremental costs (i.e. non-business as usual) costs to the target group of complying with regulation. It could be argued that Regulation (EU) No 376/2014 mainly introduced information requirements to Member States and the industry and thus, the costs incurred can be mainly classified as administrative burdens. Following a similar reasoning, the 2012 Impact Assessment for the Regulation proposal did not take into consideration compliance costs and considered all costs as administrative burdens.
At the same time, these costs can be further divided into additional categories as captured in
the
table below:
Cost categories
|
Definition
|
Direct labour costs
|
The costs of staff time devoted to completing the activities required to achieve regulatory compliance. Only the costs of staff directly involved in undertaking these activities should be included: the costs of staff supervision/management are included in the overhead cost category. Direct labour costs include two main elements: the cost of wages paid and non-wage labour costs.
|
Implementation costs
|
The costs regulated entities incur in familiarising themselves with new or amended regulatory compliance obligations, developing compliance strategies and allocating responsibilities for completing compliance-related tasks. In large part, therefore, they are short-term one-off costs.
|
Equipment costs
|
Those costs incurred by businesses whenever they need to purchase items of capital equipment to comply with a regulation. This can include both machinery and software.
|
Material costs
|
The incremental costs incurred in changing some of the material inputs used in the production process in order to ensure regulatory compliance (thus, they are sometimes called “input costs”). They are therefore ongoing costs.
|
Key methodologies for cost assessment
The methodology that inspired the approach chosen for the cost assessment is the Standard Cost Modelling (SCM), which allows to address the costs and burdens of the Regulation for both Member States and industry stakeholders. It allows the quantification of actual costs (i.e. time costs) resulting from the Regulation requirements (for Member States and industry stakeholders at MS level), to enable an assessment of the extent to which the Regulation is deemed efficient.
The SCM in its original format was designed for the assessment of administrative burdens of Member States, individuals and organizations. However, the following features were used to reinforce and complete the approach chosen:
Estimation of benefits derived from a potential reduction on the number of accidents and serious incidents
Administrative Burdens are estimated on the assumption that legislation is complied with for 100%, despite the fact that in practice, not all stakeholders fully comply with all of the requirements. While this assumption has been held for the great majority of information obligations, whenever there was certainty that a specific information requirement is not met it was not considered.
Input information and extrapolation of costs and benefits
Eighteen cost-benefit questionnaires provided the core evidence base for cost assessment analysis. These questionnaires provided both quantitative and qualitative information to shed more light on the actual costs incurred as a result of the Regulation as well as the benefits achieved.
Cost-benefit questionnaires were sent to all possible key stakeholders and targeted to national authorities on one hand, and industry stakeholders on the other. The cost-benefit questionnaire contained questions pertaining to the efficiency of Regulation 376/2014, including specific questions regarding the costs incurred as a result of the Regulation, and how these costs compare to the benefits achieved.
On the authority side, a total of fifteen (15) completed questionnaires were returned, covering fourteen (14) different Member States and EASA. On the industry side, however, only three completed questionnaires were returned: from a ground handling service provider, an aircraft parts maintenance organisation and an aviation manufacturer.
Complementing the cost-benefit questionnaires, a set of interviews was conducted with stakeholders from various operational environments and contextual settings. For the qualitative study questions these interviews provided strong evidence base for the analysis of the Regulation’s efficiency. However, in terms of quantitative information, data obtained from interviews was too limited to be used for building the basis to draw overarching conclusions for all EU Member States.
Given the challenges in data collection and the sometimes-heterogeneous responses from Member States, all this data was standardized, validated from a data consistency perspective, thus deleting potential outlier answers and checked against desk-research gathered data. Data gaps for cost driver data had to be set out by relevant assumptions, contrasted through interviews or data available in relevant literature.
Furthermore, extrapolation of quantitative information (personnel cost, training cost etc.) was challenging. Firstly, cost drivers were selected to enable extrapolation from a limited set of cases to EU level. Number of occurrences integrated into the ECR and the market size (number of organizations) were used as the main cost drivers. Secondly, data collection was not always possible at the desired level of granularity due to a combination of factors such as non-availability of data, low data quality, different reporting practices, etc. Consequently, data gaps had to be filled using methods such as cross-country extrapolation or contextual indicator comparison.