Table of contents

1    Introduction    

1.1    Context of the Web Accessibility Directive    

1.2    Purpose and scope of the evaluation    

2    What was the expected outcome of the intervention?    

2.1    Description of the intervention    

2.2    Point(s) of comparison    

3    How has the situation evolved over the evaluation period?    

3.1    Implementation of the Directive    

3.2    National monitoring, reporting and enforcement    

3.3    Current state of play and context    

4    Evaluation Findings (analytical part)    

4.1    To what extent was the intervention successful and why?    

4.2    How did the EU intervention make a difference and to whom?    

4.3    Is the intervention still relevant?    

5    What are the conclusions and lessons learned?    

5.1    Conclusions    

5.2    Lessons learned    

Annex I. Procedural Information    

Annex II. Methodology and analytical models used    

Annex III. Evaluation matrix    

Annex IV. Overview of benefits and costs    

Annex V. Stakeholder consultation – Synopsis report    

Annex VI. Additional considerations from national monitoring reports    

Annex VII. Mapping of UN Sustainable Development Goals (SDGs)    

Annex VIII. Bibliography    

Glossary

Introduction

This staff working document represents the evaluation of Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies  1 , also referred to as the Web Accessibility Directive (WAD) and its implementing acts  2 .

Web accessibility means that ‘all people, particularly people with disabilities and older people, can use websites in a range of contexts of use, including mainstream and assistive technologies; to achieve this, websites need to be designed and developed to support usability across these contexts’  3 . In the context of the WAD, accessibility should be understood as principles and techniques to be observed when designing, constructing, maintaining, and updating websites and mobile applications so that persons with disabilities can perceive, understand, navigate, and interact with them  4 . In this document, “accessibility” specifically refers to the above definition only.

1.1Context of the Web Accessibility Directive

Accessibility to information and communication technologies is an enabler of rights and a prerequisite for the full participation of persons with disabilities on an equal basis with others  5 . The main purpose of web accessibility is to remove the challenges and barriers which emerge for people with disabilities when using digital information and services on the web or in mobile applications. Accessibility – essential for some, and useful for all  6 – is therefore a key driver for the development of a knowledge-based society and makes it possible to pursue economic growth while ensuring social inclusion.

Since the internet has become a preferred way for governments to provide information and public services  7 to citizens, people with disabilities across the EU must be supported to access such information and services online. Web accessibility should be considered not only as a set of tools needed to guarantee the participation and inclusion of people with disabilities in society. It should also be considered as a fundamental means of upholding everyone’s human rights in a digitalised society, in line with the provisions of the United Nations Convention on the Rights of Persons with Disabilities (UNCRPD)  8 .

This is possible only if digital interfaces are designed, developed and maintained to ensure that they are accessible for a wide range of needs. However, as assessed in the Impact assessment accompanying the document ‘Proposal for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies’ websites’  9  (“the Impact Assessment”), before the adoption of the WAD, the level of accessibility of websites and mobiles applications in general, including those of public sector bodies, was low  10 . In addition, the market for web accessibility-related products was fragmented  11 .

In 2016, 68.1 million persons were estimated  12 as having a moderate disability and 30.5 million as having a severe disability. Taken together, these 98.6 million people accounted for about 24% of people aged 16 and above and living in the EU in private households  13 . In 2019, about a quarter (24%) of the EU population experienced long-standing activity limitations due to health problems  14 . These numbers clearly demonstrate the relevance of the issues at stake at the time of adoption.

This is the context in which the WAD entered into force on 22 December 2016, requiring the websites and mobile applications of public sector bodies (PSBs) to meet accessibility requirements. It set the first accessibility requirements in Europe for the online space. The WAD was conceived as a minimum harmonisation directive, in recognition of the fact that some EU countries had already set higher standards or might do so in the future  15 .

1.2Purpose and scope of the evaluation

This staff working document is an evaluation of the WAD and its implementing acts (the intervention), covering the first three years of its application (from the transposition deadline of September 2018 to December 2021). In line with Article 13 of the WAD, the review must take into account: (i) the Member States’ reports on the outcome of the monitoring (Article 8 refers); (ii) the use of the enforcement procedures (Article 9 refers); (iii) an analysis of the technological advancements that could make accessibility easier for excluded types of content.

This evaluation is conducted following the guidance provided by the Better Regulation Guidelines and Better Regulation Toolbox  16 . As for all evaluations, this report will assess:

·to what extent was the intervention successful and why (effectiveness, efficiency, coherence);

·how did the EU intervention make a difference and to whom (EU added value);

·is the intervention still relevant (relevance);

and will identify

·conclusions and lessons learned, to be considered in future policy developments in the area of web accessibility.

As the United Kingdom (U.K.) is no longer a Member State, it is not considered in scope of this evaluation.

This document draws on and is accompanied by an external supporting study  17 . The study analysed the evidence available from primary and secondary data sources  18 , including open and targeted consultations, in-depth interviews, a literature review and the assessment of the official national monitoring reports sent by the Member States (MS) to the Commission  19 .

Both the study and the evaluation adopted an inclusive approach to consultation activities, involving all relevant stakeholder groups affected by the WAD, with a specific focus on the participation of people with disabilities. In particular, for the first time, an easy-to-read and more accessible questionnaire offered people with disabilities the chance to take part, including those with cognitive impairment  20 .

What was the expected outcome of the intervention?

The evaluation starts with an explanation of the theoretical framework developed to assess the performance of the WAD. This framework starts from the description of (i) how the WAD was expected to work (section 2.1) and (ii) the points of comparison on which the evaluation is based on (section 2.2).

1.3Description of the intervention

The intervention logic of the WAD is presented in Figure 1 below.

Figure 1. Intervention logic of the Web Accessibility Directive  21

Problems and needs. The adoption of the WAD aimed to address four major problems in web accessibility: (i) the need to improve the functioning of the internal market providing web accessibility services, given varying legislation in MS; (ii) the very poor level of accessibility of the websites and mobile applications providing public services and information to the public; (iii) the lack of common technical standards, which created a barrier for market players and therefore for end-users; and (iv) digital and social exclusion of people with disabilities caused cumulatively by the above mentioned problems.

Regulatory fragmentation across the EU had hindered the potential growth of the market for web accessibility products and services. A wide variation in technical provisions in calls for tenders was a serious issue for web developers who therefore faced barriers in operating across borders  22 . Such variation was the result of different regulations at national – or even regional – level concerning the web accessibility of PSB websites and mobile applications. Such fragmentation was particularly problematic for SMEs, which could not bear the costs of operating in different markets requiring different standards. In addition, providers of tools for web accessibility (such as authoring tools or automatic testing tools) faced the additional barriers of having to implement all these different national specifications in their tools. This in turn caused an interoperability problem for assistive technologies, as these could not be deployed in different MS with different requirements in place. Moreover, the lack of web accessibility also required public administrations to maintain costly alternative channels, such as call centres, to ensure the provision of services to all.

In 2007, the study ‘Measuring progress of e-Accessibility in Europe’ (MeAC)  23 identified the level of web accessibility of PSB websites and mobile applications detected as considerably low. This was confirmed by the subsequent studies MeAC 2 (2010)  24 and MeAC 3 (2012)  25 . People with disabilities and older people had limited or no access to websites, and were obliged to seek potentially more costly alternatives to access information and services. The WAD therefore sought to lower these barriers.

The lack of common technical standards was one factor behind the market fragmentation described under problem one. Together with the different legal frameworks in the MS, market players faced barriers, as they could not be certain that tools acceptable in one MS would match requirements in another MS. End-users paid the price for the lack of a functioning single market, the price being lower accessibility. Finally, digital and social exclusion, is caused by a cumulative effect of the above-mentioned three problems.

Objectives. Given the above context, the WAD has two key objectives:

1)to ensure that the websites and mobile applications of PSBs are made more accessible on the basis of common accessibility requirements  26 ;

2)to harmonise the internal market for web accessibility products and services, overcoming regulatory fragmentation  27 .

Inputs. In order to achieve these two objectives, in 2016 the EU adopted the WAD, which requires the MS to ensure that the websites and mobile applications of PSBs meet common accessibility requirements introduced by the Harmonised European Standard, linked to the WAD with the Implementing Decision 2018/2048  28 . Furthermore, to ensure that provisions were implemented in a consistent way across the EU, and that MS regularly monitored the accessibility of public sector websites and mobile applications, the Commission adopted the Implementing Decision 2018/1523  29 and Implementing Decision 2018/1524  30 . These implementing decisions respectively establish: (i) a model accessibility statement (AS); and (ii) a common methodology for monitoring the conformity of PSBs’ websites and mobile applications with the accessibility requirements of the WAD.

Activities. The WAD requires the European Commission to help the MS implement the WAD. To this end, in addition to adopting the Implementing Decisions, the Commission set up the Web Accessibility Directive Expert Group (WADEX)  31 which supports cooperation between MS and with relevant stakeholders.

MS were required to assign to specific bodies the responsibilities for: (i) monitoring and reporting as required in Article 8 of the WAD; and (ii) enforcement at the national level, as required in Article 9 of the WAD. In the course of setting up national monitoring processes, MS had to involve and consult end-users with disabilities. The WAD also obliges MS to take the necessary measures to raise awareness of accessibility requirements and to promote and facilitate training programmes on accessibility.

Finally, PSBs were required by the WAD to provide and regularly update a detailed, comprehensive and clear accessibility statement (AS) on the compliance of their websites and mobile applications.

Expected outputs. Four main outputs were expected: (i) a more coherent and consistent legislative framework on web accessibility across the EU; (ii) the appointment of monitoring, reporting and enforcement bodies at national level; (iii) the publication of an AS on the websites and mobile applications of each public sector body; (iv) feedback from end users to PSBs on the accessibility of their websites or mobile applications, leading to virtuous improvement cycles.

Expected results. In line with the two key objectives mentioned above, the implementation of the WAD was expected to lead to two types of results.

First, a harmonisation of the internal market for web accessibility products and services, overcoming the regulatory fragmentation existing prior to 2016. In the identified intervention logic, the overall results were divided into three measurable outcomes: (i) increased cross-border sales of accessible information communication and technology (ICT) products and services; (ii) more companies offering accessibility expertise; and (iii) a reduction in prices for web accessibility products and services due to increased competition in the market.

Second, an increase in EU Member States and EEA countries of: (i) the number of accessible PSB websites and mobile applications (according to the WAD’s requirements) and (ii) the use of PSB websites and mobile applications by people with disabilities.

Expected impacts. The WAD aimed at increasing digital and social inclusion by improving access for people with disabilities to public sector websites and mobile applications. Greater public participation was envisaged, measured through the use of online public services and the greater ‘co-design’ of public policies (i.e. where policymakers and the public work together to design public policies). Wider impacts were also expected in the market due to legislative harmonisation and the reduction of regulatory barriers across the EU. The two most significant market results expected were: (i) a more efficient digital internal market; and (ii) web accessibility market growth, contributing to economic growth and job creation. The final expected impact of the WAD was greater digitalisation-led efficiency in the delivery of public sector services.

1.4Point(s) of comparison

The Impact Assessment estimated that the EU policies and actions available before the introduction of the WAD would not have been sufficient to solve the problems and needs explained in section 2.1. The market would have remained fragmented, and the low level of web accessibility in the public sector would have persisted. Members of the public unable to interact with online public services would have continued to experience ‘digital exclusion’ and limitations on their social, economic, and civil participation. Given that the Impact Assessment was drafted in 2012, the data gathered at the time of the initial proposal has limited relevance today, due to the rapid increase in digitalisation since then. Consequently, the evaluation re-conducted the analysis to the five main evaluation criteria suggested by the Better Regulation Guidelines: effectiveness, efficiency, coherence, relevance and EU added value.

In light of these considerations, the intervention logic described in section 2.1 explains how the inputs and activities required to meet the objectives of the WAD are expected to generate outcomes, namely short-term outputs, medium-term results and long-term impacts. These elements will be used as the main points of comparison to assess the performance of the WAD. Such points are built on the evaluation framework provided in the Impact Assessment and are directly linked to the evaluation matrix presented in Annex III:

·Point 1: relation between the observed results (medium-term) and impacts (long-term) and the objectives of the WAD. To what extent have the two main objectives of the WAD been achieved – measured through the effectiveness criterion (see sections 4.1 and 5.1.1 for the results of the evaluation). The analysis considered (i) the most relevant success factors for the application of the WAD and (ii) the gaps and challenges hindering the achievement of the objectives.

·Point 2: relation between the inputs and activities put in place for the implementation of the WAD and the observed outcomes. The costs borne to achieve the outcomes of the WAD – measured through the efficiency criterion, i.e. to what extent did the provisions of the WAD produce outputs at reasonable costs (see sections 4.1.10 and 5.1.2).

·Point 3: relation between the inputs and activities conducted to implement the WAD and the problems to be addressed. To what extent is the WAD complementary with other legislative initiatives at the European and national level on web accessibility of public services – measured through the coherence criterion, i.e., the coherence of the WAD with other legislative interventions in the area of web accessibility and inclusion, at national, international and EU level, identifying any complementarities or inconsistencies (see sections 4.1.11 and 5.1.3).

·Point 4: the difference made by the EU in achieving the outcomes of the WAD. To what extent has an EU-level intervention reached outcomes which could not have been achieved by MS on their own – measured through the EU added value criterion (see sections 4.2 and 5.14).

·Point 5: relation between the inputs and activities conducted to implement the WAD and the objectives. To what extent has the WAD been relevant to achieve its initial objectives – measured through the relevance criterion (see sections 4.3 and 5.1.5).

For each of these five points, the evaluation matrix presented in Annex III both explains the key questions set to perform an evidence-based assessment, and describes the related indicators and data sources used to judge the performance of the WAD.

How has the situation evolved over the evaluation period?

1.5Implementation of the Directive

The WAD was implemented through the following milestones: on 22 December 2016, the WAD entered into force. By 23 September 2018 (transposition deadline), MS were required to transpose the WAD into their national legislation and designate bodies responsible for monitoring and reporting functions, as well as for enforcement of the WAD. MS were to apply national legislation transposing the WAD in practice according to the following timeframe:

·from 23 September 2019 for websites published after 22 September 2018, i.e. for ‘new websites’;

·from 23 September 2020 for all other websites of PSBs, i.e. for all websites, old and new;

·from 23 June 2021 for mobile applications of PSBs.

In line with Article 8 of the WAD, all MS are required to submit every 3 years a report (the ‘monitoring report’) of the results of monitoring and enforcement procedures. The first submission was due by 23 December 2021.

At European level, in 2016 the European Commission set up the WADEX group  32 to support uniform application of the WAD at national level. WADEX is expected to achieve this goal by fostering cooperation and the exchange of best practices on web accessibility between MS and relevant stakeholders.

Three implementing decisions were adopted in 2018, to support a consistent implementation of the WAD across the EU. These implementing acts are directly applicable and need not be transposed into national law.

·Implementing Decision (EU) 2018/1523  33 laid down a model accessibility statement which PSBs must follow to ensure that their websites and mobile applications comply with the WAD (WAD Article 7 refers).

·Implementing Decision (EU) 2018/1524  34 set up a monitoring methodology and arrangements for the reporting every 3 years required from MS (WAD Article 8 refers).

·Implementing Decision on the harmonised standard (2018/2048)  35 for websites and mobile applications. This implementing decision provided for the presumption of conformity with the accessibility requirements recognised by the WAD (WAD Article 6 refers). Subsequently amended by Commission Implementing Decision 2021/1339  36 , which refers to the latest version of the Harmonised European Standard EN 301 549 V3.2.1 (2021-03)  37 .

All MS have officially transposed the WAD into national legislation  38 .

Six MS completed the transposition within the requested timeline: Slovenia, the Netherlands, Denmark, France, Italy and Spain. The U.K., still a Member State at the time, also transposed the WAD on time. The MS that did not transpose fully by the deadline can be divided into three groups:

·12 MS notified the transposition measures after the deadline: Ireland, Greece, Croatia, Cyprus, Latvia, Luxembourg, Malta, Poland, Portugal, Romania, Finland, Sweden;

·6 MS notified pre-existing measures within the deadline, without providing evidence of full compliance with the WAD: Bulgaria, Czechia, Estonia, Hungary, Lithuania, Slovakia;

·3 MS with federal systems which did not adopt measures in all regions within the deadline: Austria, Belgium, Germany.

1.6National monitoring, reporting and enforcement

MS were required to designate monitoring, reporting and enforcement bodies by 23 September 2018. The three bodies must perform the functions set out below.

Monitoring bodies must annually  39 monitor the accessibility of public sector websites and mobile applications and provide information to PSBs on how those websites and mobile applications are complying with the accessibility requirements. They may also check for the compliance with the WAD of accessibility statements published on the websites and mobile applications of PSBs.

Enforcement bodies must: (i) ensure the availability of an adequate and effective enforcement procedure to guarantee the compliance of websites and mobile applications with the accessibility requirements; (ii) investigate when a PSB affirms that making content accessible would be an excessive burden for them (the so-called “disproportionate burden”)  40 ; and (iii) ensure that feedback given by users is effectively handled (both notifications about content that is not accessible and requests for content excluded from the accessibility obligations).

Reporting bodies are required to publish in an accessible format and to submit reports on the results of monitoring and enforcement activities to the Commission every 3 years.

All Member States (MS) met the obligation to appoint these bodies. In 12 MS monitoring, reporting and enforcement are performed by the same authority, while 15 MS entrusted these functions to separate bodies. Austria, Belgium, Germany, Spain and the Netherlands assigned responsibility for enforcement to the relevant regional or local level of public administration bodies. The full list of bodies appointed by MS and notified to the European Commission is published on a dedicated web page  41 . The assignment of multiple or single function to the bodies concerned does not appear to have any impact on their efficiency and is in line with the WAD requirements.

Currently there are 25 national reports available, covering all EU Member States except France and Cyprus  42 . All the monitoring reports are published on a dedicated Commission website  43 , as well as on national websites. According to these reports, total of 10 412 websites and 298 mobile applications have been tested for accessibility by MS during the first monitoring period, in what has been described as the ‘world’s largest accessibility test’  44 .

1.7Current state of play and context

We cannot consider the period under evaluation (2018-2021), without mentioning COVID, and how it accelerated the digital transformation. The pandemic made PSB crisis information essential, and obliged use of online PSB services, to an unprecedented extent. Part of the Commission’s response has been the Digital Decade policy  45 , which sets ambitious targets for further digitalisation of the public sector.

This context helps explain some of the apparent contradictions in the public feedback, which identified progress but also the need for further improvements. To the challenge facing MS at the start, of identifying and improving accessibility of all existing PSB websites and mobile applications, were added the sudden need to digitise more, to limit, organise or replace face to face transactions, and the greater use of online PSB information and services across all government departments and levels, due to the pandemic.

The 2022 eGovernment Benchmark 2022 report found that only 16% of evaluated eGovernment websites passed all of the selected 8 accessibility criteria  46 , and expressed a need for more accessible public sector websites  47 . However, looking closer at the source data  48 and test results per criteria, 28% of websites did not fail any criterion (e.g. test passed or inapplicable). Although this pilot indicator  49 , measured with automated tools on selected criteria, does not determine compliance with the WAD, it helps determine if web content meets accessibility guidelines.

This pilot gives an insight into the state of accessibility, but the results cannot be compared directly to earlier studies (e.g. MeAC  50 ) or even national monitoring reports, due to different testing methodologies and the fact that for a complete evaluation of website’s accessibility, additional manual evaluations are needed to verify full compliance.

While administrations across Europe largely rose to the challenge and accelerated digital service transformation in the face of the coronavirus pandemic, the eGovernment Benchmark 2022 report suggested to prioritise user-centric design to ensure that eGovernment services are inclusive to users in all their diversity, including users with poor digital skills or those living with disabilities.

The 2020 Berlin Declaration on digital society and value-based digital government  51 acknowledges the public sector as an essential element for the European Single Market and emphasises the importance of digital public services in our everyday lives. According to the first progress report on the monitoring of the Berlin Declaration  52 , in 2022 the EU average score was 60% in policy area 2 on ‘social participation and digital inclusion to shape the digital world’  53 .

 The revised European Interoperability Framework  54 looks at principle of inclusion and accessibility, and observed an improvement of EU average scores since 2019  55 .

By 2021, the share of individuals using internet in the EU had risen to 90%, some 20 percentage points higher than in 2011  56 . In 2022, the share of individuals interacting online with public authorities in the EU had risen to 63% (for 25 to 64 years old) and 38% (for 65 to 74 years old)  57 from 48% and 17% respectively in 2012.

This trend is likely to continue, as explained further in sections 4.3 and 5.1.5 on relevance.

Evaluation Findings (analytical part)

This section discusses the analytical findings of the evaluation, encompassing all five evaluation criteria (effectiveness, efficiency, coherence, relevance and EU added value).

As described in section 1.2, the analysis draws from a supporting study and extensive consultation activities, including: (i) a public consultation using both a standard and an easy-to-read questionnaire; (ii) targeted surveys for different stakeholder groups; (iii) in-depth interviews conducted with relevant stakeholders; and (iv) analysis of Member States’ reports on the outcome of the monitoring and the use of the enforcement procedures (see Annexes II, V, VI).

1.8To what extent was the intervention successful and why?

The assessment of the intervention’s success level is based on the first three points of comparison described in section 2.2.

1.8.1Compliance with the Directive and conformity with accessibility requirements are not identical

At the outset, it should be mentioned that conformance to the accessibility requirements does not equate to compliance with the WAD. This distinction was sometimes overlooked, as observed throughout the evaluation, in national monitoring activities and reports, and in the wording used by monitoring tools.

For a website or mobile application to be compliant with the WAD, it must fulfil two criteria. First, it must conform to the accessibility requirements (perceivable, operable, understandable and robust), for which the Harmonised European Standard EN 301 549  58 gives a presumption of conformity  59 .

Secondly, a website or mobile application must have a detailed, comprehensive and clear accessibility statement (AS). The AS, based on the model provided by the Commission implementing decision, must include information on (i) compliance status, (ii) any inaccessible elements and (iii) accessible alternatives, (iv) how to report or request any inaccessible content, and (v) a link to the complaint mechanism if the response is inadequate. Thus, a website or mobile application without an AS is not compliant with the WAD, even if it is technically fully accessible.

Conversely, a website or mobile application that, for example, cannot be navigated with a screen reader, does not conform to accessibility requirements, as defined in the standard. However, it may still be compliant with the WAD if the inaccessible element is exempt, for example due to legitimate disproportionate burden, and this is clearly explained in the AS, with information on accessible alternatives  60 .

The box below presents the findings from the monitoring reports related to the accessibility statements.

Box 1. Analysis of findings from the monitoring reports on the accessibility statements

Inaccessible content flagged by monitoring tools may be legitimately exempt (e.g. disproportionate burden) or not in scope of the WAD  62 . The monitoring tools, especially automated ones, mostly assess the accessibility requirements set out in the WCAG (sometimes also parts of HEN), but do not consider the possible legitimate exceptions and alternatives to inaccessible content. In this context, the lack of an explicit obligation on the MS to consider the accessibility statement when monitoring a website or mobile application is not ideal.

The research literature shows that although there was already a slight improvement in the accessibility of public sector bodies’ websites and mobile applications (objective 1) in the early years following the transposition of the WAD, there is still a notable lack of accessible websites across the EU in both public and private sector  63 . However, the research in the existing literature is of limited relevance, as it is mostly based on automated accessibility checking, so excluding accessibility statements, and covers typically national level PSBs. As noted earlier, the 2007 benchmarking study Measuring e-Accessibility (MeAC)  64 found only 5% of government websites passed automatic and manual accessibility checks. In 2011, the follow-up study, using a different method, found that 34% of the EU ‘web content’ (public and private) was accessible. However, given the rapid digitalisation of the public sector since then and evolution of automatic testing  65 , these figures also are no longer particularly relevant.

Although the 2022 eGovernment Benchmark report found that only 16% of the eGovernment websites passed all of the selected web accessibility criteria, the conformance rate per criteria is more encouraging: out of all checks 76% passed, 17% failed, 7% were inapplicable  66 . Rather than giving an accessibility score per website, the modern automatic accessibility testing tools count the number of criteria that pass, fail or are inapplicable, and thus allow testers to identify conformity per criteria. For example, the eGovernment Benchmark report suggests that some accessibility criteria are better or more easily implemented than others  67 .

This is in line with the findings of the 2022 WebAIM Million  68 report. After examining the accessibility of the top 1,000,000 home pages worldwide, the report noted that while accessibility errors have decreased, no website is completely error-free.

1.8.2Progress and compliance – surprisingly hard to demonstrate

More than half of respondents (57%) to the easy to read survey  69 found government websites easier to use in the last three years, while in the standard survey 41% reported experiencing accessibility issues never or less often than three years ago. About a third of respondents in both surveys reported no changes in their experiences.

Source: Public consultation

Looking closer at content types or services offered by public sector bodies, 72% of respondents to the standard consultation observed slight or significant improvements when accessing online information (respectively 115 and 27 out of 198) over the last three years, as shown in the chart below. This opinion was shared by 52% of people with disabilities, older people, or people taking care of people with disabilities in targeted surveys.

Source: Public consultation (standard survey, n=198; not all respondents were asked this question)

As to the role of WAD, 50% organisations representing people with disabilities, older people or consumers agreed that the WAD had contributed to greater accessibility of online public services. Yet about the same number of respondents to the public consultation (57%) do not think that ‘people with disabilities can use online information just as well as others can’. This apparent contradiction can be explained – PSB web accessibility has increased from a low starting point, but persons with disabilities cannot yet access all online public services on equal terms with everyone else at a time when public services are progressively more online. The majority of respondents (79%) affirmed that their use of online public services has increased in the last three years, and 65% expected more improvements in the future.

The main accessibility issues identified when using public services online were navigation and accessing forms, followed by video content (though live time-based media is not in scope of the WAD)  70 .

From a geographical perspective, there is slight difference in experience, as measured by the surveys. More respondents in northern  71 and southern  72 European countries reported experiencing issues less often than before or never (50% in North and 54% in South) compared to central  73 and eastern  74 European countries (33% in Central and 37% in East).

One difficulty for MS in demonstrating progress in making websites and mobile applications “more accessible”, is the compliance status as defined in the Implementing Decision (EU) 2018/1523: fully, partially or non-compliant. The Harmonised European Standard EN 301 549 (HEN) sets out the means to determine a presumption of conformity with the accessibility requirements of the WAD: full compliance is achieved by meeting all the applicable requirements. Testing a website or mobile application item can thus result in: pass, fail, or not applicable. In practice this means that if any one item on a relevant page or screen fails the test, the website or mobile application can only be considered partially compliant.

Inevitably, given the wide and varied nature of these requirements, and the many different actors responsible in each MS, most of the websites and mobile applications sampled for monitoring fall in the category of partially compliant. However, partial compliance covers a wide range of situations, from nearly perfect accessibility to only just non-accessible, with no consideration of the impact of errors on the end-user. Failing some criteria (for example, no alternative text for images) would render a page technically non-compliant but usable, whereas failing other criteria (for example, a keyboard trap that prevents a user with visual impairment or limited mobility from completing a task online) would create a serious usability issue, in effect blocking access Unfortunately, there is no agreed methodology amongst accessibility experts at present, neither at EU level nor internationally, on how to quantify partial accessibility as a percentage  75 , or to weight the accessibility requirements by impact.

It should also be noted that the HEN has been updated once since 2020, and will continue to evolve. At present the HEN itself acknowledges areas for further work: its Annex D states ‘It is evident that people with limited cognitive, language and learning abilities have diverse accessibility needs and preferences and that there is a need for further guidelines and standards. Research in this area is ongoing’  76 .

Results from the national monitoring reports, however, show there is ample room for improvement in compliance as most of the websites and mobile applications monitored did not respect all of the requirements. Nonetheless, purely binary outcomes of monitoring of websites and mobiles applications (pass/fail of conformity requirements) should be taken with caution, as they do not reflect the full measure of compliance with the WAD.

According to the national monitoring reports, there were only a handful of fully compliant websites or mobile applications, but no completely non-compliant website or mobile application. This does not contradict the findings of the public consultation, as there is ample space for relevant improvements within the “partially compliant” category, without reaching full compliance.

The interpretation of such results should consider some factors. On the one hand, (i) some MS did not always follow the Implementing Decision (EU) 2018/1524  77 and defined compliance according to their own methodology, which sometimes was stricter for non-compliant websites, but also sometimes more lenient regarding full compliance. On the other, (ii) when the compliance assessment followed the Implementing Decision, the reporting of “pass” or “fail” results was inconsistent across MS, either because not applicable requirements were handled differently or because the MS also considered (and weighted) the level of breach in different ways. Therefore, it is difficult to draw a meaningful comparison between the MS from the evidence provided in the monitoring. In the second monitoring period, however, this will be addressed by requiring a recurring sample of at least 10 % of websites and mobile applications monitored in the previous monitoring period and at least 50 % that were not monitored in the previous one  78 .

1.8.3Cross-border monitoring tools are in use, although commercial tools are not perfect

As objective 2 of the WAD was to harmonise the single market, the fact that some European tools used in the monitoring exercise are being used cross-border is very encouraging.

The industry players consulted identified the most relevant drivers of success in the application of the WAD at market level as: (i) the spread of knowledge about accessibility solutions across businesses; and (ii) the increase in demand for accessibility tools.

Common technical requirements, such as the Harmonised European Standard EN 301 549, were a first essential step in removing barriers in the internal market  79 and have been useful for making public sector websites accessible. According to industry players, common requirements produced a positive impact on their business, driving demand of products and services in the national markets and an increase in opportunities to penetrate foreign markets  80 .

Several MS developed tools themselves, some of which have been also used in other countries: Belgium (Scan.accessibility.belgium.be), Finland (Salvia, using QualWeb), Luxembourg (simplA11yMonit, simplA11yGenReport, simplA11yPDFCrawler), Portugal (AccessMonitor, also used in Slovakia), and Spain (OAW Tracker, also used in Sweden).

Coupled with this, many MS used commercial tools, with at least 4 such tools used by at least 3 MS, and 2 by at least 6. More than 10 tools that help manual testing were used. For mobile applications, MS altogether used at least 8 automated tools, with a further 8 tools helping manual inspection.

All MS used (to a greater or lesser extent) automated tools in the simplified monitoring, with the exception of Croatia. However, looking at the least tested criteria, the list clearly shows that the vast majority of the automated testing tools used have been created to fit the US market, where WCAG 2.0 (published in 2008) is still relevant. Conversely, the EU legislation uses WCAG 2.1 (published in 2018) as the reference for presumed conformity in Harmonised European Standard EN 301 549. These automated testing tools can only check for 20-30% of success criteria  81 , while some leading accessibility software tools claim 57% of digital accessibility issues  82 .

For the in-depth monitoring, the list of least tested success criteria may indicate that multimedia content is either omitted from the page samples or does not feature in the websites and applications selected for monitoring  83 . This would require further investigation, as multimedia content is very important for large user groups. This issue may serve as a possible direction for further investments in the market and/ or research funding, given the expansion of accessibility requirements that will come into effect in 2025 under the European Accessibility Act (EAA)  84 .

Regarding harmonisation of the internal market for web accessibility products and services, overcoming regulatory fragmentation (second WAD objective), the evidence collected indicates that the WAD helped to improve the internal market for web accessibility services  85 . Interviews with stakeholders from the private sector  86 indicated that the WAD had lowered entry barriers to other European markets for software providers, facilitating cross-border sales. The industry players consulted agreed that both the demand and supply of web accessibility products and services had increased after the introduction of the WAD.

The WAD facilitated an increase in the number of industry actors specialising in accessibility tools in the EU market  87 . This growth of the sector is likely to further improve the internal market in the coming years. The ongoing implementation of the WAD and the upcoming EAA  88 requirements will continue to increase demand, while the in-depth interviews with the national monitoring bodies indicated that the supply of accessibility tools is still insufficient, both in quantity, and quality (not fully covering the requirements set in the HEN).

1.8.4Member States reports are difficult to compare

The first monitoring reports provide insights on the different use and interpretation of monitoring methodologies by MS.

The WAD and Implementing Decisions deliberately do not impose a particular monitoring methodology, to allow innovation and appropriate choices at national level. MS in practice both monitored and reported on different timeframes, to different versions of the technical standard, and some did not monitor the minimum sample size required. While reserving the right to pursue actions of non-compliance, it should be noted that this was the first such monitoring exercise, and came relatively soon after the practical deadlines for implementation (15 months for websites, and six months for mobile applications). In this context a range of interpretation could be expected. The most significant of these differences concerns the standard used as the baseline for presumption of conformity, with the complexity of monitoring exacerbated by the difficulties encountered in interpreting the guidelines of the WAD. Work will continue to ensure this is fully understood, especially as this standard will be further revised to reflect the requirements of the European Accessibility Act.

Although some MS monitored a smaller sample size  89 , all MS who reported carried out both simplified (all but one using automated tools) and in-depth monitoring. Only 2 MS have not reported to the Commission  90 .

Some MS committed to test more than is required, according to analysis of findings from the Member States going beyond the scope of the WAD in the monitoring exercise  91 . For example, Germany also tested three requirements, on sign language, easy-to-read language, and the ‘UE’ standard for PDF, not covered by the HEN. Both Portugal and Finland checked WCAG 2.1 AAA  92 criteria in their monitoring exercise, although not an obligation under the WAD. The results were not shared in the monitoring report. On the procedural side, in Greece it is obligatory to publish self-assessment reports.

From analysis of the reports, the following points emerge:

·It also emerged in the interviews that many MS are unaware of the accessibility criteria set out in the Harmonised European Standard EN 301 549 (HEN), and so they only focused on WCAG 2.1 criteria for monitoring. 16 MS monitored only WCAG success criteria, omitting the clauses 5, 6, 7 and 12 of the EN 301 549 (HEN). Only four MS monitored all the relevant requirements of the HEN. This is a significant problem, as while Annex A of the HEN sets the criteria for the presumption of compliance, some MS may use different baselines to judge conformity. Several MS mentioned that the requirements of WCAG 2.1 and HEN for mobile applications were hard to interpret, making it difficult to assess compliance.

·MS conducted the monitoring in different periods within the set timeframe. Implementing Decision (EU) 2018/1524 asked for the first monitoring report to perform the monitoring ‘within’ the period between 1 January 2020 and 22 December 2021 for websites, and 23 June 2021 and 22 December 2021 for mobile applications. In five countries  93 , the monitoring exercise was conducted only in 2021. While the legal text is open to different interpretations, it should be noted that all MS conducted monitoring “within” the period defined in the WAD.

·The scope of the monitoring exercise varied. Under the WAD and Implementing Decision (EU) 2018/1524, MS should select a sample of websites and mobile applications  94 that (i) is as representative as possible (considering the different administrative levels, the type of services provided by the public sector, and the geographical distribution); (ii) is proportionate to the population of the country, and (iii) takes into account the views of stakeholders, in particular organisations representing persons with disabilities and older people  95 .

·The differing interpretations of standards, accessibility conformance rules and testing methods can lead to inconsistent or even conflicting results. This makes it difficult or even impossible to reliably compare monitoring results produced by different tools. To address uniform interpretation, for example, of the WCAG and provide consistent results, the W3C has produced a recommendation – Accessibility Conformance Testing (ACT) rules, which are constantly evolving  96 .

It should be noted that Bulgaria reported that iOS-based applications are not used by a significant number of people with disabilities, because no adequate speech synthesiser is currently available in Bulgarian for that platform.

During the interviews, several MS stated that for the next monitoring period they would welcome clearer guidance on how to report results of the monitoring activities and a standard template at the EU level. This could substantially improve the outcomes’ reporting comparability and results’ analysis. Another concern that emerged in the interviews is the high burden that the current monitoring process imposes on monitoring bodies, which often do not have enough expert staff to carry out these activities.

When asked about their reasons for not monitoring all requirements, some monitoring agencies stated they were not to be aware of the current requirements, and others said that the external experts hired to carry out the audits did not have the skills to do so. One possible reason for the difference in scope may be that the automated tools used to support the testing cater to WCAG and do not contain all the HEN requirements. The Commission had flagged this as a topic for discussion in the WADEX meetings and has published detailed information about this topic online  97 .

Some MS reported that their current list of digital public services is incomplete and/or unreliable as mapping and counting all public sector websites and mobile applications is impractical, if not impossible, particularly given the many different administrations involved at national, regional and local level. This limits the MS’ capacity to consider the entirety of websites and mobile applications in scope of the legislation  98 .

1.8.5Member States have committed to training and awareness-raising

Article 7(4) of the WAD states that MS should facilitate training programmes and awareness-raising activities on the WAD and its main provisions, asking for the relevant information in a dedicated section of the monitoring reports. All MS which delivered the report provided this information, with the exception of Ireland and Portugal. The monitoring report from Lithuania only shared the country’s plans for future training in this area. The remaining 22 countries stated that they had implemented the provisions of Article 7(4) mainly through three channels: (i) setting up dedicated events and workshops at the national and sub-national level 99 ; (ii) organising training programmes for different stakeholder categories  100 ; and (iii) drafting technical guidelines and materials  101 .

The analysis of the literature reveals several other factors influencing the adoption and implementation of web accessibility standards in PSBs  102 . These include: (i) assigning clear responsibilities; while stakeholders claim to be in full control over their part of the process, once confronted with non-conformance of the end result, they pointed to others for responsibility; (ii) management decisions; in municipalities where managerial commitment to the website and online services is high, stakeholders perceive fewer problems; (iii) perceived benefits; if management does not perceive sufficient benefits from better accessibility, priorities move away from accessibility; and (iv) legislation; although it does not directly guarantee the proper implementation of accessibility standards, it has a positive influence on the process.

Opinions differed over the involvement of people with disabilities in the implementation of the WAD obligations. Several engagement activities and procedures were launched by national authorities to consult relevant stakeholders about the implementation of the WAD. Some MS even set up permanent committees with representatives from DPOs, industry, academia and public authorities. Furthermore, many of them involved end-user groups and relevant organisations in the process of selecting websites and mobile applications to be tested for the first monitoring activities, an obligation under the Implementing Decision  103 . The involvement of people with disabilities ranged from consulting on strategy and advising on the mechanisms implemented by the WAD to being involved in user testing, providing the views and knowledge of the group benefitting most from the proper implementation of measures and monitoring, following the principle “nothing about us without us”.

The facilitation of training and awareness raising in MS can be considered a success factor in the implementation of the WAD. Since the beginning of the monitoring period, a variety of training initiatives have been promoted by most of the MS. Several countries introduced training on accessibility topics, mostly targeting employees in PSBs. For instance, Czechia is launching a new course for PSBs on web accessibility, which will provide a certificate to participants who complete the course. Some MS (e.g. Slovakia and Spain) reported that new training courses were devised based on the results of the monitoring activities.

However, a 2019 survey conducted by the European Disability Forum  104 showed significant dissatisfaction with the level of engagement from relevant stakeholders. Respondents criticised the lack of consultation by MS before the adoption of the national laws and denounced the poor guidance and lack of support for stakeholder organisations. In addition, the results of the WAD review survey targeting DPOs show that 60% of DPOs (29 out of 48) reported little or no involvement in either the implementation of the WAD or in the selection of websites and mobile applications to be monitored.

1.8.6Feedback and enforcement procedures under-reported and under-used

The Implementing Decision (EU) 2018/1524  105 requires MS to describe and detail their use of the enforcement procedure set up at the national level, to be used when initial feedback has failed. MS may also (but are not obliged to) give data on the feedback received by the PSBs through the first-level feedback mechanism laid down in Article 7(1)(b) of the WAD. On the enforcement procedure, 18 monitoring reports describe the procedure in place, while only two countries (Germany and Finland) reported quantitative information on the complaints they had received. The enforcement procedures of various MS include the possibility of using administrative fines for PSBs that do not comply with the WAD  106 .

In general, countries did not give any quantitative information about the feedback mechanism, apart from Finland, which shared results from a survey it had conducted on the mechanism. On the issue of feedback, several countries filled in the section devoted to feedback mechanisms with information that seemed unrelated. Some countries reported only on complaints or on feedback given to the monitoring body by PSBs or experts. Such shortcomings in the reports suggest that the optional reporting requirements on feedback mechanism were either unclear or left room for interpretation to the MS.

In addition, from the public consultation it appears that both these procedures are little used.

While the lack of complaints  107 could be seen as very positive (e.g. no accessibility issues found), among the reasons reported for not complaining were the inaccessibility of feedback forms due to use of the online ‘captcha’ tools and a perception that a complaint would not change anything. Some individuals who had complained reported negative experiences, as enforcement bodies did not always reply satisfactorily to their complaint, nor seem to be aware of the mechanism. This lack of complaints, however, should not be fully interpreted as an indication of a malfunctioning instrument. It may also be the case that the answers given or actions taken in the feedback mechanism, are satisfactory in not giving rise to further formal complaints.

The awareness of these mechanisms should continue to be monitored, as further research on the needs and barriers for end-user possible feedback and complaint may be needed, given the current low level of use.

The box below presents the findings from the monitoring reports on the enforcement procedures.

Box 2. Analysis of findings of the monitoring reports on the enforcement procedures

1.8.7Accessibility experts are needed, as job opportunities exist

MS highlighted the lack of accessibility expertise both internally in the public sector and available to hire. This was a particular problem when MS needed to hire external auditors (testers) for the monitoring activities. The public consultation also identified the most relevant challenges as: (i) PSB web managers often lack skills in web accessibility; (ii) technical difficulties in making existing websites/mobile applications compliant; and (iii) a lack of resources (both staff and money).

The WAD has shown beneficial effects in stimulating the job market for web accessibility services  109 and capacity building. This has created new job opportunities and profiles (e.g. web accessibility specialist, web accessibility coordinator, web accessibility software engineer, web accessibility analyst)  110 , and changed existing ones (e.g. the work of front-end engineers, user-interface developers, etc.). These findings are also confirmed by the available literature on the topic  111 , as testing for accessibility requires distinct skills, as well as the use of specific tools.

However, despite the increased number of private service providers in the EU after the implementation of the WAD, MS monitoring bodies reported a shortage and/or high cost of experts to conduct accessibility audits. A Wall Street Journal article  112 from September 2021 cited LinkedIn data that the number of job listings with “accessibility” in the title had grown 78% in the year August 2020 – July 2021. In September 2022, the EURES job database listed over 1,000 job vacancies with accessibility in the title  113 . As manual accessibility checks must be done locally in the local language(s) it is important to ensure there is sufficient accessibility expertise in all regions in all the MS. 

The accessibility industry representatives interviewed expect that in the coming years the job market will develop even more, driven by higher demand from PSBs and by the new legal obligations for the private sector set out in the European Accessibility Act  114 .

Digital accessibility therefore needs to feature far more prominently in information technology courses, as well as usability design training. For more than a decade, the Digital Competence Framework for Citizens (DigComp)  115 , the EU-wide framework for developing and measuring digital competence, has provided a common understanding, across the EU and beyond, of what constitutes “digital competence”.  In 2022, the Digital Competence Framework – DigComp 2.2  116 – was updated to explicitly mention ‘digital accessibility’ across several knowledge, skill and attitude competencies, including the awareness of legal requirements such as accessibility statements. This should ensure accessibility is recognized as a basic part of digital competence and is taught throughout the MS.

This fits also the targets and policy of the Digital Decade which is about making digital technology work for people and businesses  117 and with the target of 20 million ICT specialists in the EU by 2030  118 , with significant funding under the DIGITAL programme to bridge the digital skills gap  119 . There is clearly a high need for digital skills when it comes to integrating digital technologies in the work processes, including digital accessibility skills for both civil servants, and for contractors. Skills and competences of civil servants could be systematically boosted through specialised training  120 .

One should emphasize the opportunities for people with disabilities in this growing job market, particularly given the surge in remote work  121 , which may help some.  In most EU countries, the percentage of remote jobs has increased nearly 5 times compared with the pre-COVID period, especially in the Information & Communication Technology sector  122 . People with disabilities could be involved as professional accessibility auditors (testers) of websites and mobile applications, which can be stepping-stone jobs into the workplace. This is an important issue, as the unemployment rate for persons with disability is significantly and persistently higher than that of the general population  123 .

MS also identified a lack of official skill certification systems  124 . Currently, the IAAP (International Association of Accessibility Professionals)  125 offers the main internationally recognised professional qualification, with four types of certifications for online accessibility. In practice, auditors and external experts are generally hired based on their professional experience and references they can provide, rather than on professional qualifications.

1.8.8Accessibility is increasing, but more is needed, especially at regional and local levels

As the public sector continues to digitise its delivery of public services, the expectation increasingly is for the public to use online services by default, also in line with the Digital Decade  126 target to have 100% of key public services online by 2030. According to targeted consultations organisations representing persons with disability, consumers and older people, the general opinion is that the WAD has been more effective at improving the accessibility of PSB websites at the national level, and, to a lesser extent, at the regional and local level in all MS  127 . 51% of these respondents believed that, since the application of the WAD, there has been an improvement in online information available on websites at national level. The statistics suggest a slightly lower impact on the accessibility of websites at the regional (45% reported an improvement) and local level (38%).

This may be related to the size of public administrations, as larger PSBs tend to have more budget, staff and experience to enforce web accessibility. It is hoped and expected that as PSBs progress to the digital decade targets, accessibility will be embedded by design from the start – e.g., as observed in national recovery and resilience plans that feature prominently measures for digitalising public services and introducing or improving e-government solutions to modernise public administration. To close the digital divide in rural and remote areas, the EU supports investments in an unprecedented manner. For example, around EUR 16 billion Recovery and Resilience Facility (RRF) reforms and investments have already been approved to roll out digital connectivity networks in the next four years, especially in rural regions  128 .

On the adoption of similar standards in the private sector, as encouraged by Recital 34 of the WAD, web accessibility has been – and still is – a low priority in the private sector  129 (see below Box 3). Some of the examples mentioned at Recital 34 will be covered by the EAA. Industry players reported that the WAD, together with the EAA, should help change this situation, with the scope of accessibility obligations expanded under the EAA, so introducing incentives for private actors to develop accessibility solutions (see section 4.1.11).

In this context, many MS went beyond the scope of the WAD in their national laws, as detailed below.

Box 3. Analysis of findings from the Member States going beyond the scope of the WAD  130

1.8.9Technological advancements could make accessibility easier for content types not covered by WAD

Article 1 of the WAD allows for the temporary or permanent exclusion from its scope of some types of content, some websites, and some mobile applications. Technological advancements in digital accessibility could make accessibility easier for content types currently excluded from the scope of the WAD due to lack of automated or efficient and easy-to-implement means at the time of adoption of the WAD  133 . Analysis on the technologies that have significantly evolved in the last years shows that automation through artificial intelligence and machine learning is increasingly used in tools relevant for web accessibility  134 .

Respondents to the public consultation  135 identified the following six technological advances as being the most relevant for the WAD:

·assistive technologies driven by artificial intelligence (AI);

·accessible authoring tools;

·AI and machine learning applied to monitoring;

·accessibility monitoring and testing tools;

·the ‘internet of things’ for accessible products and services; and

·biometrics for identification and security aspects.

The experts interviewed agreed that automation is increasingly being used in different tools relevant for web accessibility. Based on the level of maturity, relevance for the WAD and size of the affected user groups, the study prioritized 13 technologies for deeper analysis (e.g., speech or sign language recognition, automated accessibility testing, web authoring tools) in scientific fields such as machine learning, natural language processing, deep learning, computer vision, etc  136 .

To highlight some of the most relevant, the market offers automatic captioning for videos or online videoconferencing tools, built on speech recognition technologies, but only in major languages. While the quality of captions (subtitles) is incrementally improving, so is language coverage. This is a welcome development for WAD, as accurate automatic captions in official EU languages (and others) will make it cheaper and easier to subtitle live time-based media, currently excluded from the WAD. This will help PSBs to comply with the requirement to subtitle pre-recorded time-based media, which are in scope of the WAD. There are promising developments in automated text simplification and summarisation  137 to support people with cognitive disabilities or reading difficulties  138 . Research continues into digital access to sign languages  139 , and speech / text multimodal conversion  140 .

While all promising, there is not sufficient maturity in all languages to modify the current exemptions. Except for English and, to a certain extent, German, Spanish and French, at present all European languages are massively under-resourced in terms of data, tools and technologies  141 .

It should be noted that the context is evolving in favour of accessibility outside the WAD obligations. Social media platforms, for instance, make some such tools available by default. This evolving context is changing expectations and raising awareness, which may have a positive impact also on PSB websites and applications even before any potential revision of the WAD. Given the likely spill-over effect of the increased accessibility requirements of the EAA from 2025 onwards, there appears to be no pressing need to revise the WAD exceptions in the near term.

1.8.10Efficiency, adequacy and proportionality of the resources

This section presents evidence-based analysis on (1) cost-effectiveness of the WAD and (2) adequacy and proportionality of the resources.

Overall, the supporting study suggests that the WAD was designed and implemented efficiently, and the use of EU resources was adequate and proportionate  142 .

However, a detailed assessment of economic, social, and environmental impacts is limited, mainly due to the difficulties in quantifying and monetising web accessibility, inclusion and non-discrimination, as well as due to the limited availability of relevant data. The social impact of the activities for persons with disability in the digital sphere cannot be measured in the absence of comprehensive inclusion and equality data.

The main costs imposed by the WAD are those for PSB to comply with the accessibility standards and those for the monitoring activities borne by MS. However, it is difficult to measure such costs, due to a lack of quantitative data.

The evidence collected shows that accessibility related costs have not constituted a major barrier to implementing the WAD, as the national monitoring reports show that there was limited use of the disproportionate burden exemption clause.

The benefits can therefore be mainly measured in qualitative terms. According to the supporting study the main economic benefits are an increase in the demand for web accessibility products and services in the national markets, driven by PSBs, and an increase in opportunities for accessibility service providers to penetrate foreign markets.

Source: Public consultation (standard survey, n=311)

With little information publicly available about the costs, it is not surprising that the majority of respondents (58%) were neutral on whether the objectives of the Directive had been achieved at a reasonable cost (out of 311 respondents, 81 neither agreed nor disagreed, 101 were not sure). However, a majority of the respondents (184 or 59%) agreed with the statement that the benefits arising from the availability of accessible websites and mobile applications outweigh the costs of implementing the WAD.

According to 50% of the PSBs consulted (31 out of 62) in a targeted survey, the main benefit of the WAD has been the possibility of providing services to a larger pool of people because of improved accessibility. Moreover, consulted PSBs appreciated the role of the WAD in encouraging organisations to provide clearer accessibility requirements in procurement procedures. Some of the stakeholders interviewed identified as benefits increased usability and fewer complaints about services.

68% of the PSBs consulted believe that the WAD introduced additional economic costs for their organisations. The main sources of recurring costs were: (i) improving the accessibility of digital services; (ii) training staff; and (iii) drawing up accessibility statements. Despite this, only a few MS reported that the WAD imposes high costs on PSBs to implement its requirements. In fact, MS reported only very few cases of using the disproportionate burden exemption clause.

The monitoring bodies provided information on the costs related to monitoring activities. However, the information provided and the figures varied significantly, which makes it difficult to provide a high-level overview across all MS. The supporting study contains some of these figures. Costs were reported only by few MS and range from a minimum cost of €65,000 up to a maximum cost of €650,000. The costs of additional activities, such as awareness-raising and training, were not reported by the MS  143 .

According to the industry players replying to the targeted survey, the WAD introduced new costs, associated mainly with: (i) hiring new specialists; (ii) updating the products and services to meet the new accessibility requirements; and (iii) expanding internal capacities to meet the new, higher volume of demand. However, according to 60% of industry players, the common accessibility requirements introduced by the WAD had a positive impact on their business. The main benefits highlighted by the respondents were an increase in the demand for web accessibility products and services in the national markets and an increase in opportunities to penetrate foreign markets. The stakeholders interviewed reported that there is still a lack of industry players offering accessibility services. However, they expect a significant increase in service companies and organisations working in this area in the coming years. This represents an opportunity for the technology and educational sectors as new digital skills will be required by the market. The Digital Skills and Jobs Platform  144 may be an important EU-wide instrument to support companies looking for specialists.

Finally, the supporting study also suggests that EU funds invested in the implementation of the WAD were adequate in promoting research on web accessibility and accessible technology, and in the deployment of new solutions, as well as in supporting accessibility and social inclusion for persons with disabilities  145 .

The EU supports MS through a wide range of instruments, from strategic and operational analysis to the design and the financing of reforms and investments. As mentioned in section 4.1.8, the EU supports investments in digital transformation of the public sector in an unprecedented manner  146 .

According to a recent Commission report to the European Parliament and the Council  147 , measures supporting the digitalisation of public services and e-government represent more than a third of the digital expenditure financed by the Recovery and Resilience Facility (RRF). The share of ‘E-government, digital public services and local digital ecosystems’ in RRF expenditure supporting the digital transformation is 36% (EUR 47 billion)  148 . A third of that amount (around EUR 16 billion) has already been approved to roll out digital connectivity networks in the next four years, especially in rural regions. Moreover, the EU aims to also leverage connectivity investments through the new Cohesion Funds, the EAFRD, InvestEU and EIB loans, and, last but not least, through CEF Digital  149 . Also the Digital Europe Programme recognizes that digitalisation can facilitate and improve barrier-free accessibility to all, including older people, people with reduced mobility or with disabilities, and people in remote or rural areas  150 .

It is worth noting that public administrations at central, regional and local level implement directly about 35% of the EU budget. In some MS, more than 50% of public investments were funded under the EU cohesion policy  151 .

These investments in e-government, and the digital transformation in general, should directly improve web accessibility. The Common provisions regulation  152 explicitly spells out that operations must comply with applicable Union law and national law, which includes WAD and its transposed rules in national law. The EU has co-financed many national projects through its structural funds, which have produced several relevant outcomes, both in terms of new tools/solutions and in knowledge sharing  153 . These can be considered a return on investment for the achievement of the two main objectives of the WAD.

The Quality of Public Administration Toolbox  154 is a summary of Commission knowledge on improving public administration, which has a dedicated knowledge base on digitalisation and accessibility. The Technical Support Instrument (TSI)  155 aims to provide tailor-made expertise (technical support) on the ground to help MS design, develop and implement growth-enhancing reforms in a wide range of policy areas, including for digital public administration  156 .

The Commission also provides knowledge services like the Knowledge4Policy platform that brings together policymakers and scientists across the EU for evidence-informed policymaking  157 . These centres support policymaking by monitoring the uptake and impact of artificial intelligence in the public sector  158 .

To illustrate some more detailed examples related to WAD, the EU has allocated resources to implement the WAD in two further ways. Firstly, the WADEX group, using it to facilitate the exchange of best practices between the MS, as well as with relevant stakeholders and experts where appropriate. Secondly, the European Commission funded, through the Horizon 2020 programme, pilot projects and preparatory actions, such as (e.g. WAI-Tools  159 , WADcher  160 , WAI-Guide  161 , and We4Authors  162 ) in the areas of web accessibility, accessible technology, and the deployment of solutions. Most of the MS considered that their engagement in the implementation of the WAD had contributed to achieving its objectives. The WADEX group has been very useful for these stakeholders, especially by making possible to obtain and exchange technical information and to meet and engage with other people working on web accessibility. Many member appreciated the meetings as a forum to exchange views on the standards, the monitoring methodologies and other practical examples.

1.8.11Coherence with other policies and instruments

The supporting study confirmed that the WAD is coherent internally, and with relevant EU and international policies and actions, as well as with the UNCRPD, the EU Charter of Fundamental Rights and the European Pillar of Social Rights. This is based on feedback provided by stakeholders during consultation activities and the review of relevant EU and international policies.

The supporting study shows that the WAD is coherent internally with the overarching Commission strategies on disability, the European disability strategy for 2010-2020  163 and the Strategy for the rights of persons with disabilities for 2021-2030  164 . The current strategy itself refers to this evaluation and commits to include accessibility and inclusiveness in the reinforced EU digital government strategy, focusing on human-centric and user-friendly digital public services across Europe that respond to the needs and preferences of European citizens, including the needs of persons with disabilities.

Effective use of digital technologies requires the removal of accessibility barriers for persons with disabilities and investing in their digital skills. As announced in the Digital Education Action Plan 2021-2027  165 , MS will be supported in securing assistive technologies and in providing an accessible digital learning environment and content. To promote disability-inclusive education, MS can use the opportunities offered by EU funding, including the Erasmus+ and the European Solidarity Corps programmes, setting out dedicated inclusion measures. Furthermore, for schools and educational buildings, MS can also address accessibility through the Renovation Wave  166 aiming to make liveable environments accessible to everyone.

The European Accessibility Act (EAA)  167 complements the WAD in both scope and coverage. It has a strong digital relevance that aims to improve the functioning of the internal market for accessible products and services, by removing barriers created by divergent rules in MS. As the WAD, the EAA establishes accessibility requirements for a range of products and services that have been identified as rather important for persons with disabilities while having diverging accessibility requirements across EU countries. This includes, among others, e-commerce, banking, and transport services, concerning both public and private sectors.

The MS had to transpose the EAA by 28 June 2022 and apply in practice as of 28 June 2025, so businesses across the EU need to ensure that the services they provide and products they deal with comply with the EAA’s accessibility requirements. The scope EAA is complementary to WAD, covering also websites and applications of the private sector. Presumption of conformity with the EAA will be based on the same (updated) European standard EN 301 549 harmonised under the WAD for websites and mobile applications. In addition, two other updated standards (EN17161 and EN17210) and three new ones will be developed to support specific elements of the EAA.

The WAD is also coherent with other relevant EU legislation in the field of digital policies. Most recently, the updated set of rules for all digital services within the Digital Services Act  168 also contains an obligation for the Commission to encourage and facilitate the drawing up of Codes of Conduct. Such a code will ensure that services provided are designed in an understandable and robust way in order to ensure they are accessible to persons with disabilities, and it is publicly explained how accessibility requirements are met. Providers of services as well as civil society representatives and recipients of the services should be involved in their drafting. The Audiovisual Media Services Directive (AVMSD)  169 obliges MS to adopt rules to impose accessibility requirements on providers of audiovisual media services. The European Electronic Communications Code (EECC)  170 , which aims to modernise EU telecoms rules, covers service providers and requires that the services they provide be accessible. EECC does not cover content and does not lay down accessibility requirements for what is placed on the internet, but aims for equivalent access for end-users with disabilities, addressing the need for available and affordable equipment and services and the access to emergency services. The Directive  171 and the Regulation implementing the Marrakesh Treaty  172 share the same objectives of accessibility as the WAD but cover different types of products. The Marrakesh Treaty sets the legal framework for changes needed in national copyright regimes to implement legal copyright exemptions for the benefit of blind, visually impaired, and otherwise print disabled persons that would enable the beneficiaries or authorised entities to make accessible format copies of books and other written material. The Treaty also facilitates the exchange of accessible format copies across national borders between authorised entities or with the beneficiaries. Finally, the eIDAS Regulation  173 , which ensures that people and businesses can use their own national electronic identification schemes (eIDs) to access public services available online in other EU countries, is also coherent with the objectives of the WAD, by requiring that trust services and end-user products used in the provision of those services shall be made accessible for persons with disabilities, where feasible. Accessible eID and trust services are highly relevant for completing processes on the web, important in public services covered by the WAD, so accessible eID and trust services should lead to increased use of such online PSB services.

The requirements in the WAD together with the measures above establish a coherent regulatory system that facilitates digital accessibility both in the public and private sector.

Almost all monitoring bodies interviewed during the consultation activities reported no overlaps or inconsistencies between the WAD and other relevant national measures related to web accessibility. German national law also sets another standard for checking the accessibility of PDF files (the ‘PDF UE’ standard), but the two standards, although having different requirements, do not contradict each other  174 .

The WAD fits into the aims and objectives of major treaties and instruments addressing issues of digital and social inclusion for people with disabilities, to which the Commission is party, both at European and international level, i.e. the EU Charter of Fundamental Rights  175 and the European Pillar of Social Rights (EPSR)  176 , the UNCRPD  177 and the UN Sustainable Development Goals (SDGs)  178 .

The WAD is coherent with the relevant articles of the EU Charter of Fundamental Rights, including: (i) on increased social participation for older people and people with disabilities (Articles 25 and 26); (ii) on the right of access to documents (Article 42); (iii) on removing any form of discrimination, including disability (Article 21); and (iv) on freedom of expression and information (Article 11).

Regarding the EPSR, coherence with the objectives of the WAD is evident especially: (i) on inclusion of people with disabilities, including through services that enable them to participate in the labour market and in society (Principle 17); and (ii) on access to essential services which states that everyone has the right to access essential services of good quality, including digital communications (Principle 20).

The WAD is also consistent with and supports the proposed digital principle that everyone should have access to technology and to all key public services online across the Union  179 .

The WAD has significant synergies with the UNCRPD, currently the most relevant piece of international law in the area of disability and is coherent with its objectives. Its two most relevant articles are: Article 9, which focuses specifically on accessibility, and Article 21, which focuses on freedom of expression and opinion, as well as on the freedom to retrieve and access information. As society becomes more digitalised and most administrative services are available online, a number of other UNCRPD articles show synergies with the WAD  180 . As a precondition of human rights, accessibility indirectly links also to the exercise of rights of persons with disabilities under the UNCRPD provisions  181 .

Finally, all of the UN Sustainable Development Goals (SDGs) are coherent with the achievement of more inclusion of people with disabilities in society  182 . The provisions of the WAD and its implementing acts are, in particular, relevant for achieving the objectives of SDGs 1, 3, 4, 8, 9, 10, 16, and 17 as flagged by the UN Flagship Report on Disability and Sustainable Development Goals  183 . Achieving such goals requires the accessibility of PSBs’ digital administrative procedures, including services of general interest. The list of all relevant SDGs is provided in Annex VII with a description of how the key provisions of WAD and its implementing acts contribute to these  184 .

1.9How did the EU intervention make a difference and to whom?

As described above, especially at sections 4.1.7 and 4.1.8, the WAD has directly impacted persons with disabilities, accessibility solution providers, as well as the public sector itself.

The analysis considered to what extent the value of the EU intervention can be considered additional to what could have been achieved by MS acting alone. In particular, it sought to assess to what extent the WAD made a difference in removing internal market barriers and in improving digital access to persons with disabilities and older people. Finally, the analysis evaluated to what extent the WAD generated additional value compared to national measures implementing the UNCRPD, the EU Charter of Fundamental Rights and the EPSR.

Source: Public consultation (standard survey, n=311)

Most of the respondents to the public consultation agreed that there is an EU added value of the WAD in improving online accessibility for public services and in increasing the digital inclusion of users. Although information pre- and post-Directive on the accessibility of PSBs’ websites and mobile applications is limited, interviewed monitoring bodies reported a significant increase in the awareness of web accessibility among PSBs since the implementation of the WAD. In particular, the monitoring activities have been an important occasion for awareness raising. Many PSBs addressed accessibility problems after the start of the monitoring period.

The other added value reported by monitoring bodies is that, for 22 MS, the adoption of the WAD meant the introduction of national laws on web accessibility that did not previously exist. The general finding from the in-depth interviews was that the MS alone would not have reached this level of awareness and web-accessibility related activity without the WAD.

Concerning the EU added value in removing barriers across the internal market, most respondents across different stakeholder categories agreed that the main benefit has been the harmonisation of national legislation, which has brought tangible results in the internal market, particularly the common standard which facilitates cross-border provision of web accessibility products and services. Industry players noted that several barriers to the cross-border provision of web accessibility services had been removed. Most PSBs consulted (79%, 49 out of 62) believed that the WAD had contributed to making national laws on web accessibility requirements more harmonised. Similarly, 72% of the organisations consulted, representing persons with disabilities, older people and consumers (35 out of 48 such organisations consulted) agreed that the WAD had contributed to the harmonisation of the legislative framework on web accessibility across EU countries. In addition, 54% of the organisations also believed that the WAD enhanced cross-border cooperation among different associations of people with disabilities.

The main contributions of the WAD to the national initiatives of MS have been: (i) increasing the demand for web accessibility tools from public authorities (reported by 65% of the respondents); and (ii) spreading awareness about accessibility solutions among web professionals (reported by 62% of the respondents). 61% of industry players agreed with the statement that both the demand for – and supply of – web accessibility products and services increased after the introduction of the WAD. In addition, 44% of the respondents emphasised the impact of the WAD in incentivising private actors to develop accessibility solutions.

The available literature agrees that the introduction of harmonised standards is a first essential step in removing barriers in the internal market. In addition, some authors have also shown the beneficial effects of the WAD in stimulating the job market for web accessibility services  185 .

1.10Is the intervention still relevant?

The supporting study and the online surveys find that the WAD is still relevant, particularly considering the evolving European and international policy context  186 .

Source: Public consultation (standard survey, n=311)

As can be seen above, the overwhelming majority of respondents to the public consultation consider it still relevant for PSBs to provide accessible content and services both online (97%) and on mobile devices, including apps (95%). It is still highly relevant for WAD to contribute to digital inclusion (97%) and to reduce regulatory differences among MS in the field of web accessibility (87%).

The WAD remains relevant for all main stakeholders, as the increasing use of PSB websites and online services shows the continuing need for more accessible websites and mobile applications. The amount of online information provided by public sector organisations has increased in the last three years according to 85% of respondents (265 out of 311). Similarly, most respondents reported an increase of interactive services (84%), mobile applications (78%) and online documents (72%)  187 .

PSBs overall have not reacted to the new accessibility obligations by limiting their online presence. The surveys and interviews noted only a few examples of limiting online presence, such as reducing the number of published pdf documents, removing originally live-streamed videos after 2 weeks, or publishing videos on third-party/social media platforms. Overall, MS have not widely reduced online content in order to avoid the WAD’s requirements.

Indeed many MS have gone beyond the scope of the WAD in their national laws, as detailed at section 4.1.8, which implies a genuine commitment to digital inclusion for persons with disabilities.

Regarding the policy context, the WAD objectives are directly relevant to the implementation of the UNCRPD. On the UN policy framework, the WAD objectives are also still relevant to several global SDGs  188 , including SDG 10 on reducing inequality. The WAD objectives are still pertinent to the Charter of Fundamental Rights of the European Union.

The fact that the WAD does directly not apply to EU institutions was considered a limitation by some of the stakeholders interviewed. The Commission has already announced measures to address this in the new Strategy for the rights of persons with disabilities 2021-2030  189 , and has adopted an action plan on web accessibility to ensure its web presence is compliant with the provisions of the WAD  190 .

Such intervention is also relevant in the aftermath of COVID-19 which highlighted how important it is that people with disabilities can access public sector online crisis information on equal terms  191 .

Tackling the digital divide is even more important, given the historic shift to using digital technologies in work, education, communication and public services  192 . During the COVID-19 pandemic, digital interaction has been the public administration’s main (and often the only) communication channel with citizens, enabling it to ensure business continuity.

The responses to the public consultation show that almost all respondents considered the objectives of the WAD to still be relevant today  193 . Up to 68% of people with disabilities or older people reported using online public services of national, regional, and local PSBs at least monthly. Looking at changes since the adoption of the WAD, 50% of the respondents (142 out of 311) to the public consultation affirm that their use of government websites increased slightly (23%) or significantly (27%) in the last 3 years. Regarding mobile applications, 78% agreed that the number of applications provided by PSB has increased. On the kind of public services used online, health services were identified as the most accessed service (12%), followed by money and tax (8%), utilities (8%), benefits (7%), employment (7%), disability (6%), and citizenship services (6%).

In terms of scope, almost half of the respondents (49%) to the public consultation (151 out of 311) agreed that the WAD adequately covers online public services to ensure the full participation of people with disabilities in digital society. On subject and content types not covered by the WAD, most of the respondents favoured removing the current exemptions and/or expanding the scope of the WAD to other areas. For example, full inclusion of universities in the requirements of the WAD was supported by 81% of respondents, while full inclusion of schools was supported by 70%. There was also strong support for full inclusion of NGOs (56%), online mapping services (57%), live videos (53%), extranets/intranets (51%) and third-party content (47%) given technological advances.

Finally, on the Harmonised European Standard EN 301 549 (v2.1.2), many respondents (63%) to the public consultation consider that the standard has been useful for making public sector websites accessible. Around one third of the respondents (32%) agree that the standard still covers all relevant end-user groups, with half of respondents (51%) not sure or neutral on the matter. In the interviews, a few MS mentioned that services for people with cognitive impairments should be better covered.

What are the conclusions and lessons learned?

1.11Conclusions

The adoption of the WAD in 2016 aimed to address two major problems in web accessibility: the need to improve (i) the functioning of the internal market for the provision of web accessibility; and (ii) the poor level of accessibility of websites and mobile applications providing public services.

All MS integrated the requirements of the WAD into national legislation and embarked on activities: (i) to make PSB websites and mobile applications more accessible; (ii) to ensure monitoring, reporting and enforcement of measures; (iii) to promote training programmes for PSBs on the accessibility of websites and mobile applications; and (iv) to raise awareness of both the accessibility requirements introduced by the WAD and the possibility for users to give feedback if they encounter cases of non-compliance with the accessibility requirements.

1.11.1Effectiveness

Since the adoption of the WAD in 2016, progress has been made concerning both the accessibility of PSB websites and mobile applications The WAD is going in the right direction towards the achievement of its objectives, as both cross-border demand and supply of web accessibility products and services increased after the adoption of the WAD. Since PSBs still face challenges in complying with the requirements established by the WAD, there is still room for improvement.

The Directive has helped to improve the internal market for providing web accessibility services. Most importantly, the harmonisation of national legislation throughout the EU has substantially improved the opportunities for private actors offering web accessibility products and services. Given this harmonisation, the WAD has lowered barriers to the supply of web accessibility services and products in other countries within the European market and facilitated the increase of cross-border provision of products and services for web accessibility.

There are no official skill-certification systems in most MS, which makes it more difficult to identify appropriately trained experts and to assess the specific expertise of the auditors. Monitoring bodies therefore generally assess the expert’s professional expertise based on former experience, rather than formal qualifications.

The Directive is also working to improve the level of web accessibility of public sector websites and mobile applications across the EU. The results from the monitoring reports show there is much progress to be made in making all public sector websites and mobile applications fully accessible to persons with disabilities. This is partly due to the wording of the WAD, calling for ‘more accessible’ websites or mobile applications, which can comply with the WAD, even if they do not fully meet all the accessibility criteria of the Harmonised European Standard. This should be read since the definition of “partial compliance” is extremely broad. Regardless, many positive considerations should be made.

Monitoring has been considered an important success factor in achieving the first objective of the WAD, as many PSBs addressed accessibility problems identified during the monitoring activities, so showing the value of monitoring and reporting exercises. The publication of the monitoring results is important, since PSBs can learn from each other’s reports. Moreover, comparing results can be very useful to help allocating resources, as they provide a benchmark and raise awareness of the need to “catch up” on accessibility.

Transparency of monitoring results can encourage healthy competition between, for example, municipalities and/or regions, where no one wants to have the lowest score. Good and improving scores can be used for various communication purposes. Suppliers of ICT services (including web developers, producers of authoring tools, and other service providers) can feature satisfied clients in their marketing activities. Equally, suppliers whose clients have bad results should quickly try to improve. Companies with expertise in accessibility issues and developers of accessibility tools can use the reports to focus their efforts on solving specific issues. PSBs, and organisations representing people with disabilities (DPOs) can use the reports in their awareness-raising campaigns, and academic research can explore and support accessibility efforts.

The interviews with MS authorities highlighted that the implementation of the WAD at national level, together with the training and awareness-raising activities promoted by MS had helped to significantly raise awareness among PSBs at different administrative levels. Additionally, the testing activities during the monitoring stage have been an additional important occasion for awareness raising. In some cases, the number of websites with accessibility statements doubled from the first to the second year of the WAD’s implementation.

Similar conclusions cannot be drawn yet on mobile applications, given that the monitoring period for applications was very short in the first reporting period, and did not allow time for re-checking findings.

The trends noted are likely to continue based on the infrastructure now in place. However, most of the monitoring bodies agree that it is still too early to notice an increase in the level of accessibility as a specific result of the monitoring activities. More comparable evidence will become available starting the second monitoring period, which requires a recurring sample of websites and mobile applications monitored in the previous monitoring period. Given this context, the level of web accessibility is expected to continue increasing in the coming years: accessibility is a process, not a one-off exercise.

1.11.2Efficiency

The results from the public consultation, and the lack of complaints about the cost of implementing the WAD, allow positive conclusions to be drawn as to the efficiency of the WAD.

Both from the perspective of PSBs and of monitoring bodies, the WAD introduced additional economic costs to their organisations. However, the overall costs of the monitoring activities conducted were reported only by a few MS and are all very different in nature. The supporting study contains some of these figures. The overall costs for the monitoring activities range from a minimum cost of €65,000 up to a maximum cost of €650,000. There are no available figures on the costs of making websites and mobile applications compliant with the WAD’s requirements.

The main benefit of the WAD, according to the results of the consultations, has been to provide services to more people thanks to improved accessibility. In addition, PSBs can achieve direct cost savings by improving web accessibility. The most common example of cost savings for PSB are related to the use of digital services, compared to printed documents, phone services or face to face consultations (note: no data was available on cost savings, which could be requested on a voluntary basis for the next reporting period).

Most of the consulted stakeholders believed that the WAD had not (yet) been effective in reducing the prices of these accessibility services. Despite an improvement in the internal market, many MS highlighted both: (i) a continued lack of experts in the market; and (ii) experiencing problems when they needed to hire external auditors for the monitoring activities. The current lack of relevant technical expertise means there is a lack of competition, keeping prices relatively high, meaning that the single market does not yet function as expected. However, a significant increase in the number of service companies and organisations working in this area is expected in the coming years, due to the impact of the EAA. Cross-border provision of tools is available and can be expected to increase.

1.11.3Coherence

On the coherence of the WAD with other legislative interventions in the area of web accessibility and inclusion, at the international, EU and national level, the findings have not identified any relevant inconsistencies or overlaps. On the contrary, there are several areas of complementary interaction between the EAA and the WAD, both in scope and coverage. For example, the EAA covers: general-purpose computer hardware and operating systems; automatic teller machines (ATMs); ticketing machines; check-in machines; various types of consumer terminal; audiovisual media services; transport services for air, bus, rail and waterborne passengers; banking services; e-books; and e-commerce. The web accessibility requirements related to this complementary scope are harmonised with the requirements of the WAD.

1.11.4EU added value

The EU added value of the WAD has been proven from the evidence collected. The main added value identified has been the harmonisation of national legislation, which has brought tangible results in the internal market.

For most countries, the adoption of the WAD also meant the introduction of national laws on web accessibility that were not present before, and reference to the same European standard. As a result, industry players interviewed have noted that several barriers have been removed and the internal market has substantially improved. The interviews carried out with MS, show that since the implementation of the WAD there has been a significant overall increase in the awareness of web accessibility among PSBs.

1.11.5Relevance

The WAD is still considered as relevant for the current policy context, at all levels: national, EU and international.

The WAD remains highly relevant for its main stakeholders, as the amount of online information provided by public sector organisations has increased in the last three years, a trend expected to continue in line with the Digital Decade target to have 100% of key public services online by 2030. The EU’s commitment to the digital transformation and tackling the digital divide is even more important, given the historic shift to using digital technologies in work, education, communication and public services since the current Commission took office. It was further highlighted during the COVID-19 pandemic, when digital interaction has been the public administration’s main (and often the only) communication channel with citizens, enabling it to ensure business continuity.

The context is evolving in favour of accessibility beyond the public sector (and scope of the WAD), as the Digital Services Act introduces the possibility of a dedicated code of conduct addressing the specific needs of people with disabilities and the practical application of the European Accessibility Act from 2025 onwards.

1.12Lessons learned

From the evaluation of the Directive, these are the main lessons learned.

·Insufficient accessibility expertise and shortage of professionals. Many MS reported a lack of basic skills in web accessibility, within the PSBs as well as in the market. The problem is not limited to the availability of technical experts to test accessibility; website design and graphic design sectors also need awareness raising and remain hard to reach on current channels. This means that PSBs: (i) have difficulties in ensuring they have the necessary expertise to make their websites and mobile applications accessible; (ii) find it hard to procure accessible solutions; (iii) are unsure about how to formulate their requirements to the market and (iv) find it hard to procure auditing and testing services. This leads to two further problems: (i) it is difficult for market players to deliver suitable websites or mobile applications; and (ii) many PSBs lack the skills to check that the resulting services they have bought are compliant with the requirements of the WAD.

·Furthermore, there is a lack of accessibility experts who can support PSBs in the area of testing and auditing. This gap is expected to be partly filled by an increase in knowledgeable market players in response to increasing demand. However, the pace of market growth has been slow so far. Therefore, the efforts already made by MS to launch training and capacity-building activities in the area of web accessibility should be strengthened (see below under ‘Training and support’). It would also be beneficial to ensure that professional certification is used when hiring or recruiting web professionals. Forward-looking, increased offer of basic accessibility training for ICT (information and communication technology) curricula in universities would help further address the shortage. Moreover, as end user testing is encouraged in the monitoring methodology, there may be more opportunities for employment for persons with disabilities. Finally, the EU’s ambitious ‘Digital Decade’ target of increasing the number of ICT professionals from 8 to 20 million by 2030 offers a unique opportunity for web accessibility to be made a common component in all relevant digital courses.

·Accessibility-related costs do not constitute a main barrier to implementing the WAD. Despite most PSBs stating that the WAD introduced additional economic costs, the evidence collected shows that accessibility related costs do not constitute a main barrier to implementing it.

·Little use made of the feedback mechanism. In the next years it is important to continue to raise awareness publicly and ensure that this mechanism is enforced at the national level, giving all end-users the possibility to provide feedback.

·Enforcement is not yet reaching its potential. Even though all MS have appointed their enforcement bodies and set up formal enforcement procedures, the effect of these bodies and procedures seems to be limited so far. On other aspects of enforcement, little evidence was found about: (i) starting a procedure on the initiative of the enforcement body itself; (ii) reviewing disproportionate-burden exemption claims; or (iii) connecting enforcement measures to the monitoring activities.

·Training and support – share best practice. Most MS launched training and awareness-raising activities after the introduction of the WAD. However, the activities reported in monitoring reports are often described in a very broad and generic way, making it difficult to identify and exchange best practices that could be replicated. Given that these activities are considered one of the main success factors in the implementation of the WAD, more information and more exchanges between MS on best practices may improve these instruments.

The analysis of the monitoring reports reveals some challenges related to the monitoring exercise itself. Overall, the monitoring and reporting mechanisms under the WAD significantly limit the comparability of the results achieved in the MS. More specifically, the most relevant challenges could be identified in the bullet points below.

·The flexibility of the monitoring methodology provided in the Implementing Decision has been beneficial for MS with previous knowledge of web accessibility. However, in some cases it has paved the way to unintended consequences.

·Some MS do not involve end-users in the testing phase, whereas others rely solely on end-users to carry out the testing. In the former case, there is a high risk of missing important aspects of web accessibility (especially since the accessibility expertise of users is often lacking). In the latter case, results can be biased due to the abilities of testers (for example, blind users cannot check design-related aspects).

·Some MS asked PSBs to conduct the in-depth monitoring on their own websites and applications without ensuring a clear process to validate the results.

·Reporting on some aspects that this evaluation finds as key elements is optional under the legislation and so rarely included (e.g. accessibility statements, feedback mechanisms, and use of the disproportionate-burden clause).

·In this context, the lack of an explicit obligation on the MS to consider the accessibility statement when monitoring a website or mobile application is not ideal.

·There are different interpretations of the reporting and sampling requirements. Some MS have used the sample of websites initially chosen for in-depth monitoring in their simplified monitoring as well. They have usually done this as a quality-assurance process for the findings of the automatic tools used. This approach is consistent with the WAD, but as a result, the total number of websites monitored could be lower than expected, and therefore fewer PSBs may be confronted with accessibility issues detected. Furthermore, conformance testing tools and methods do not necessarily produce comparable results due to differing interpretations of requirements. Some MS have reported the results from the different monitoring methods in an aggregated way, not distinguishing between detecting non-compliance and verifying compliance (simplified and in-depth testing respectively). This obscures the results of the different methods and makes the exercise less useful.

·As requested by several monitoring bodies consulted, one possible way to address these practical issues could be to launch joint work of the European Commission and MS on (i) a clear reporting structure to increase the coherence of the reporting among MS; and (ii) voluntary guidelines to orient MS in reporting on the accessibility statements, feedback mechanisms and use of the disproportionate-burden clause. This would make the monitoring exercise more efficient for reporting authorities and more consistent across countries. In addition, it would make it easier to compare results and measure progress over time.

·Minimum requirements established by the WAD are still unclear at national level. Monitoring reports reveal a clear lack of awareness of the minimum requirements in some cases. In the interviews, some MS claimed to be unaware of the minimum requirements listed in Annex A of the Harmonised European Standard EN 301 549 (HEN). Other MS revealed that they focus only on WCAG 2.1 criteria and do so for three main reasons: (i) these are the ones covered by the chosen automatic tools; (ii) these criteria are the ones known by the experts; or (iii) these criteria are considered easier to handle. As a result, Clauses 5, 6, 7, and 12 of the HEN have sometimes been omitted from monitoring. The lack of knowledge persisted although the topic has been covered in WADEX meetings and that relevant information has been published on the Europa.eu portal. More clear communication at EU and national level on the monitoring requirements (which may further evolve) is required for future monitoring and reporting exercises.

·MS need support and guidance in addressing the challenges, gaps and inconsistencies identified in the monitoring exercise and in the supporting study accompanying this evaluation. The guidance could take the form of written guidelines, workshops or other initiatives. This guidance could aim to increase the awareness and capabilities of MS in complying with their obligations under the WAD, and strengthening correct and complete application of the WAD at all administrative levels. As ensuring accessibility is a process, strengthening correct and complete application of the WAD at national level will help achieve its objectives. Embedding accessibility in digitalisation and public communication strategies in public sector helps not only reach higher accessibility compliance rates, but also maintain it over time.

·There are diverging reporting practices and differences in the interpretation of compliance status. Some MS have not always followed the compliance-status categories laid down in the Implementing Decision, whereas others drew up compliance rules according to their own methodology. It is, therefore, not possible to compare results between MS. Additionally, even when the compliance status is calculated in line with the Implementation Decision, the reporting of ‘pass’ or ‘fail’ is not coherent among MS (as explained in section 4.1.1 above). So the results do not reflect the level of accessibility in a common and consistent way.

·MS would need support and guidance on uniform definitions of the various compliance statuses if the reporting process is to produce more useful and comparable results.

·Finally, it should be noted that the MS that used assistive technology to complement their testing activities used almost exclusively screen readers. This makes the monitoring biased towards one of the nine user categories covered by the WAD and its implementing acts  194 , namely users without vision.

Annex I. Procedural Information

1.Lead DG, Decide Planning references

This evaluation was led by Directorate General Communication Networks, Content and Technology (DG CNECT), Unit G3 ‘Accessibility, Multilingualism and Safer Internet’.

The DECIDE reference is: PLAN/2020/8486.

2.Organisation and timing

On 2 September 2020, an Interservice Steering Group (ISG)  195 involving services with an interest in the measure under evaluation, was created to steer the evaluation in line with the ‘better regulation’ guidelines.

The ISG was consulted over the course of the evaluation, in conjunction with key milestones and deliverables, using various channels (virtual meetings, written exchanges, bilateral meetings, collaborative workspace).

In October 2020, DG CNECT issued a request for services and terms of reference for a tender on external study supporting the review of the application of the Web Accessibility Directive.

In November 2020, DG CNECT published the Evaluation Roadmap  196 for the initiative on the ‘Have your say’ portal, describing purpose and scope of the evaluation. The roadmap was open to public feedback for two months.

The study contract was signed on 25 January 2021, with task duration of 16 months. It began with a kick-off meeting on 12 February 2021 and ended in June 2022 (see Annex II).

In June 2022, the ISG, as well as disability coordinators from additional services  197 were consulted on this draft staff working document.

3.Exceptions from the better regulation guidelines

No exceptions from the usual procedural requirements of the ‘better regulation’ guidelines were required or applied to this evaluation.

4.Consultation of the Regulatory Scrutiny Board

Not applicable.

5.External expertise

The evaluation was supported by an external consortium composed of PwC EU Services EESV (PwC), Open Evidence (OE) and, as a sub-contractor, Funka Nu AB (Funka). Due to corporate restructuring, one of PwC members, PricewaterhouseCoopers Public Sector Srl, changed its name to Intellera Consulting Srl (Intellera) in July 2021 and continued to service the contract as the lead contractor under the new name.

The consortium collected the evidence, performed the study tasks, and submitted deliverables in accordance with the terms of reference. The final validation workshop on the findings and key results of the study was held on 20 May 2022 as part of the Global Accessibility Awareness Day and the European Month of Diversity  198 .

The draft Final report of the ‘Study supporting the review of the application of the Web Accessibility Directive (WAD)’ was submitted by the consortium on 15 May 2022.

In addition, DG CNECT regularly consulted the Web Accessibility Directive Expert Group (WADEX)  199 composed of Member State experts and invited relevant stakeholders. A total of 18 online meetings have been held in the period from January 2021 to June 2022 covering topics relevant for the review of the WAD. The latest meeting discussing the review findings was held on 28 June 2022.

6.Evidence, sources and quality

Most of the evidence was collected with the support of an external contractor (see ‘Study supporting the review of the application of the Web Accessibility Directive (WAD),’ Final Report, PwC, Intellera Consulting, Open Evidence, Funka, September 2022). The study used a mixed-method approach, combining both primary and secondary data-collection activities.

The evaluation also considers evidence from Member State reports  200 on the outcome of monitoring and enforcement procedures under the WAD  201 .

The evidence is based on external expertise and knowledge from Member States’ competent authorities, PSBs, international and umbrella organisations, independent bodies dealing with accessibility, NGOs and organisations representing persons with disabilities and older people, ICT (Information and Communications Technologies) industry players and providers of accessibility products and services, academic and certification bodies, accessibility experts, and citizens, especially persons with disabilities, and older people, through the numerous consultation activities held by the Commission and the contractor (see Annexes II and V).

The quantification of the costs and benefits of the intervention was in some cases difficult, due to unavailability of data in Member States (see section 4.1.10 and Annex IV). The analysis of the effectiveness and efficiency criteria was thus limited, as described in Annex II, section 7.

Nonetheless, it was possible to draft conclusions based on alternative proxy data and the triangulation of findings from the literature review and the consultation activities with stakeholders (section 5).

The overall evaluation is considered robust and thorough.

Annex II. Methodology and analytical models used

The evaluation has been mainly based on a supporting study provided by an external contractor. The study adopted a mixed-method approach, combining both primary and secondary data-collection activities.

The external contractor reviewed and analysed information available in the literature, including academic papers and official reports (see section 1) as well as the first national monitoring reports shared with the European Commission by 23 December 2021. In addition, the contractor set up a consultation strategy to involve the most relevant stakeholder groups affected by the application of the Directive at the European and national level. The strategy planned the following activities:

·an open consultation – namely the public consultation launched by the European Commission – held from July to October 2021 (section 2);

·four targeted surveys open between October and November 2021 (section 3);

·in-depth interviews conducted between January and February 2021 (section 4).

1.Literature review

The literature review was based on a consolidated theoretical approach, also defined as scoping review. This kind of review attempts to provide an initial indication of the potential size and nature of the available literature and tends to privilege breadth over depth of coverage for a clearly defined specific focus.

On geographical scope, the external contractor focused on Europe, although it also took into consideration as potential benchmark models papers from other countries that were deemed relevant.

The selection of documents to be analysed was based on several criteria: (i) number of references, (ii) type of publication, (iii) year of publication (within the last two decades) (iii) language (English-language was prioritised). Once a first group of relevant literature was defined, a snowball approach was used to extract further relevant documents.

Overall, the literature review included not only academic papers, but also ‘grey’ literature identified online, such as such as policy documents, official reports, industry publications and statistics, working papers, published policy briefs or conference papers. As a result, the contractor revised over 60 documents. The Commission analysed further documents for the purposes of this staff working document (see bibliography in Annex VIII).

2.Public consultation

The public consultation was open from 19 July to 25 October 2021. The aim of the public consultation was to collect views and experience on the impact of the Directive on the accessibility of public sector body websites and mobile applications from the widest possible audience. It was based on one questionnaire with between 40 and 66 questions (depending on the category of stakeholder responding to the questionnaire), and the questions were divided into two different sections: (i) general questions, meant for all the respondents; and (ii) specific questions, meant for stakeholders with a knowledge of the Directive. In both sections, additional questions were available only for specific stakeholder groups.

To complement the traditional format of EU survey, respondents also could reply to an easy-to-read version of the questionnaire. This easy-to-read questionnaire was comprised of 10 questions to give the public, especially those who may have found it difficult to reply to the long version, the opportunity to provide feedback on key aspects of the Directive (e.g. on their level of satisfaction when using public sector websites or mobile applications and on their use of the feedback mechanism).

Before publishing the official questionnaire of the public consultation, 11 pilot interviews were conducted with different categories of key stakeholders to test the questions and see how accessible and easy to understand they were.

Particular attention was given to ensure a sufficient geographical coverage, in order to avoid that the overall analysis resulted biased towards the opinion of respondents from few countries. This potential risk was firstly mitigated by involving key umbrella organisations at the European level. Moreover, targeted follow-up email or phone calls were made by the external contractor with relevant stakeholders in the least represented countries. Further information on the results of the public consultation is provided in Annex V.

3.Targeted surveys

Targeted surveys were conducted on four key stakeholder groups to complement the evidence gathered through the public consultation, by asking more specific questions to: (i) public sector bodies; (ii) people with disabilities, older people, and those taking care of people with disabilities and/or older people (formal and informal carer); (iii) Organisations representing people with disabilities (DPOs) and older people; and (iv) industry players and technology providers.

These stakeholder groups were selected in order to cover different perspectives on the effects of the Directive:

·people with disabilities, older people, and those taking care of people with disabilities and/or older people (formal and informal carer), as well as organisations representing people with disabilities (DPOs) and older people were asked to provide their opinion on the extent to which the Directive (i) has affected web accessibility of public information and services in EU Member States, and (ii) has helped to increase social and digital inclusion of persons with disabilities in the EU;

·Public sector bodies were asked to provide their opinion on the main costs and benefits brought by the Directive at different governance levels (national, regional and local) and relevant evidence on the current state of implementation of the Directive in the EU Member States;

·Industry players and technology providers were asked to provide their opinion on the extent to which the Directive had an impact on the web accessibility market in the EU and to identify technological advancements that had emerged in the European market since the Directive was applied

These questionnaires comprised both open-ended and closed-ended questions. The surveys were developed in English only and remained open according to different timespans as set out in the table below.

Table 1. Targeted survey audience and dates

For the public consultation, particular attention was given to ensure a sufficient geographical coverage, to avoid that the overall analysis results being biased towards the opinion of respondents from a few countries. This potential risk was firstly mitigated by involving key umbrella organisations at European level. In addition, the external contractor contacted relevant stakeholders in the least represented countries to encourage participation. Further information on the results of the targeted surveys is provided in Annex V.

4.National Monitoring reports

The external contractor also conducted a systematic analysis of national monitoring reports delivered to the Commission by the appointed reporting authorities of each Member State, in line with Article 8 of the WAD. Many Member States submitted the monitoring report to the Commission after the deadline set out in the Directive (i.e. 23 December 2021), and two have not yet reported. Accordingly, 25 national reports were analysed, covering all EU Member States excluding France and Cyprus.

All monitoring reports are publicly available in a dedicated webpage  202 and were provided in national languages. When an English version was not provided by the Member State, the European Commission has provided a courtesy translation into English using its automated translation services.

5.In-depth interviews

To fill data gaps, in-depth interviews were conducted to complement the information collected in the literature review, the public consultation, the targeted surveys, and the national monitoring reports. These in-depth interviews covered the effectiveness, efficiency, relevance, coherence and EU added value of the Directive. Interviews were carried out by videoconference and were based on a semi-structured questionnaire shared in advance with interviewees.

Table 2. Number of interviews, by stakeholder group

Further information on the results of the in-depth interviews is provided in Annex V.

6.Final validation workshop

The findings and recommendations that emerged from the supporting study were discussed in a final workshop. The workshop aimed to validate the key results presented in the supporting study. It was held on 20 May 2022 as part of the 11th Global Accessibility Awareness Day and the European Month of Diversity.

Over 450 people registered for the event with participants from more than 7 different stakeholders groups including public sector bodies, organisations of people with disabilities, accessibility experts, representatives from universities or research, ICT generalists, private sector and non-governmental organisations (NGOs). There was also a good balance between participants from both EU (from 26 different Member States) and countries from outside of the EU (from 24 non-EU countries). The most represented non-EU countries were: Norway, USA, U.K., Canada and Australia.

7.Methodological Limitations

The interpretation of the evaluation findings presented in section 4 above should consider the following methodological challenges:

·Limited evidence on the actual impacts of the Directive. The transposition process of the Directive at the national level was planned to close by 23 December 2018. This means that the implementation of the Directive in Member States started in 2019, namely only two years ago. In addition, many Member States were not able to meet such deadline, and this means that the timeframe of implementation in some EU countries turned out to be even shorter. As a result, in some cases, limited evidence is available to assess the impact of the Directive at national level.

·Limited availability of data. This evaluation comes at a very early stage, with Member States sharing for the first time the monitoring reports on 23 December 2021. Thus, the amount of data available is limited and difficult to compare. For example, quantitative figures available on costs experienced by stakeholders is extremely limited.

·Limited comparability of monitoring results. National Monitoring Reports provided by Member States to the Commission resulted difficult to compare in many sections as sometimes monitoring results were reported in different ways (depending on the tools used) or some sections were not filled in, leading to incoherent outcomes among Member States. This may be due both to translations mistakes (Reports provided in non-English language were translated with English machine translation) or to different interpretations of the contents to be provided in some sections.

·Geographical distribution. Overall, the amount of evidence collected in some countries is considerably higher compared to other countries. Despite the great effort made by external supporting study team, the response rates for the different stakeholder consultation activities are uneven across countries.

·Stakeholder representativeness. The results of the public consultation and of targeted surveys shall not be considered as statistically representative of each stakeholder group, since (i) the consultation strategy was not based on sample design techniques and (ii) some stakeholders engaged through dissemination did not reply to questionnaires or surveys. Therefore, the interpretation of the findings should consider that:

oIn the long version of the public consultation, a limited number of replies was received from human rights networks, ombudspersons, and older people organisations, while replies were mainly received from standardisation and certification bodies.

oThe targeted survey for individuals with disabilities, older people, and people taking care of persons with disabilities and/or older people (formal and informal carers) gathered a few replies from carers (both in the case of older persons and in the case of persons with disabilities).

oThe targeted survey for organisations representing persons with disabilities, older people or consumers gathered evidence mainly from organisations representing persons with disabilities. Consumer organisations and older people provided a limited number of answers.

oThe targeted survey for public sector bodies covered 22 monitoring, reporting or enforcement bodies coming from 15 different countries.

The issues listed above may limit the analysis especially in relation to the effectiveness and efficiency evaluation criteria. However, it was possible to draft conclusions based on the triangulation of findings from the literature review and the consultation activities with stakeholders.

Annex III. Evaluation matrix

The table below summarises the evaluation matrix, which is structured around the five evaluation criteria and respective evaluation questions. Indicators used to address each question were largely based on those developed in the SMART 2017/0068 study. Some relevant indicators were also added in order to address all the five evaluation criteria.

Annex IV. Overview of benefits and costs

The table below summarises the costs and benefits of the implementation of the Directive. A discussion of the efficiency of the Directive, as well as of the limitations of the data collection related to the costs, has been presented in section 4.1 of the evaluation report. Information in this annex is based on interviews and surveys, extracted from the supporting study, and only reflects the views of the authors of the supporting study.

Annex V. Stakeholder consultation – Synopsis report

1.Introduction

This document summarises the consultation activities held by the European Commission for the review of the Directive (EU) 2016/2102  203 (Web Accessibility Directive or WAD). The consultations aimed at collecting evidence from different stakeholders to evaluate the implementation of the Directive.

2.Consultation activities

The consultation activities were conducted between July 2021 and February 2022, including:

·A public consultation from July to October 2021  204 ;

·Four targeted surveys between October and December 2021;

·39 in-depth interviews conducted with relevant stakeholders between January and February 2022  205 .

The public consultation and the targeted surveys were conducted online  206 , and widely disseminated among stakeholders.

The overall consultation strategy was constantly updated to fill gaps emerging both in terms of geographical coverage and representativeness of stakeholder groups. The in-depth interviews were conducted using videoconference platforms (Microsoft Teams and Zoom).

2.1Public consultation

The aim of the public consultation was to collect evidence on the impact of the Directive and its implementing acts, especially to what extent these:

1)have made it easier for people with disabilities to access public services and information, strengthening social and digital inclusion;

2)are still relevant and fit for purpose, considering related laws and changes in technology; and

3)have harmonised the web accessibility market.

The standard survey included from 40 to 66 questions, depending on stakeholder type, structured around the five formal evaluation criteria: effectiveness, efficiency, relevance, coherence, and EU added value of the Directive. For the first time, respondents also had the opportunity to answer to an easy-to-read version of the questionnaire, not requiring log-in (hereinafter also referred to as the “easy-to-read survey”). The questionnaire included 10 questions and offered a simpler and more accessible way to give feedback on key aspects of the Directive (e.g., on the level of satisfaction when using public sector websites).

Overall, the two versions of the public consultation received 967 replies from stakeholders and end-users. The easy-to-read survey attracted more than twice as many responses (656) as the standard survey (311).

Source: Public consultation

Replies were received from 32 countries – all EU Member States, EFTA/EEA countries (Norway, Switzerland, Liechtenstein), as well as the United Kingdom and Albania. The largest number of responses were submitted from Sweden, Italy, Netherlands, Finland, and Germany.

This distribution is important when analysing the results, as 60% of the responses come from five countries where web accessibility policies have been in place long before the adoption of the WAD. 18% of the responses come from Sweden  207 .

Source: Public consultation

Considering the standard survey, the most represented stakeholder groups were citizens (78) and public sector bodies (65), accounting for 46% of the total respondents, followed by a relatively high number of web accessibility experts (55) and organisations representing persons with disabilities (40).

Source: Public consultation

Persons with disabilities were especially targeted in this consultation, as among citizens they are the main beneficiaries of the web accessibility directive. Nearly 40% (354 out of 967) of all respondents considered themselves as a person with a disability or having a recognised disability. Among them, about 80% responded to the easy-to-read survey (278 out of 354), showing that the easy-to-read survey could capture views which the standard survey alone probably would have missed.

Source: Public consultation

2.2Targeted surveys

The additional, targeted surveys focused on four different categories of stakeholders: (i) Public sector bodies, (ii) Industry players and technology providers, (iii) Organisations representing persons with disabilities, older people or consumers, (iv) Individuals with disabilities, older people, and people taking care of persons with disabilities and/or older people. The questionnaires included a mix of open and closed questions, addressing all the evaluation criteria, and were conducted between October and December 2021.

2.2.1Public sector bodies

The sample of public sector bodies consisted of 62 respondents, representing mostly public authorities (50%) and monitoring, reporting or enforcement bodies (35%). The most represented countries were Sweden, (22%) Germany (13%) and Poland (10%).

2.2.2Industry players and technology providers

The survey targeting industries received 52 responses, with different types of expertise. In particular, 44% of the respondents worked as expert consultants in topics related to accessibility, 11% were ICT system providers, 10% worked in institutions providing web accessibility training. In terms of size of the organizations, the sample represented a heterogeneous group of companies, from small to large enterprises. The most represented countries in this area were the Netherlands, Sweden, Denmark, Finland and Belgium, representing roughly 50% of the sample.

2.2.3Organisations representing persons with disabilities, older people or consumers

The sample of NGOs consisted of 48 respondents, mostly working in organizations representing people with disabilities (88%). In addition, a small share of the sample (8%) represented consumer organizations and other bodies (4%). The most common disabilities of the people represented by the organizations were with visual or hearing impairments, speech impairments, mental/intellectual disabilities and neurological impairments. Most of the organizations operated at the national level (65%) while a minority worked at regional (6%) or local (13%) level. A small proportion of organisations (16%) reported working internationally. The most represented countries in this area were Sweden (20%), Belgium (10%) and Slovenia (8%).

2.2.4Persons with disabilities, older people, and people taking care of persons with disabilities and/or older people

Overall, 40 people took part in the survey. Sweden and Italy were the most represented countries, with 38% and 15% of respondents respectively. 22 people reported having some form of disability. 11 people reported living with people with disabilities, 16 people worked on accessibility issues and 12 were part of organizations representing people with disabilities.

2.3In-depth interviews

39 interviews were conducted. The in-depth interviews included 28 authorities covering 21 Member States, as not all accepted the interview or replied to the request of interview, and other relevant stakeholder groups. The Table below shows the number of interviewees per stakeholder group. List of interviewees is annexed in Tables 3 and 4 in this Annex V, section 4 below.

Table 2. Number of interviews, by stakeholder group

3.Main results of the consultations

3.1Use of online public services

The initial responses to the standard survey of the public consultation aimed to gather views on the impact of the Directive on (i) access to online public services, (ii) users’ experience when dealing with such services and on (iii) knowledge and use of the feedback and complaints mechanisms. With regard to these three aspects, evidence was also collected from the easy-to-read survey, the results of which are explicitly mentioned when relevant.

3.1.1Access to public services

The open public consultation investigated the kind of public services accessed online, and the ways used to access them, through a specific sub-set of questions targeted only to respondents identifying themselves as citizens, members of organisations representing persons with disabilities or older people, web accessibility experts or in the residual category other. A total of 198 responses were submitted by respondents in these categories.

The majority of respondents (187 out of 198) declared to make a frequent use of domestic online public services, ranging from 39% of weekly users, to a 33% of daily users and a 22% of monthly users. Percentages are significantly lower when considering the use of public services of another EU country, given that 71% of respondents (142 out of 198) replied to use such services less than once a month (41%) or never (30%). With regard to the kind of public services used online, health services are the most accessed (150 out of 198), followed by money and tax services (106), utilities services (99), benefits services (92) and employment services (86).

The results of the open consultation reveal that when accessing public services online, the most important accessibility issues were experienced with navigation and forms (reported by 50% of the 198 participants replying to this question) and issues related with image and text contents (40%). This is partially confirmed by the responses received to the easy-to-read survey, where navigation (41%) and forms (17%) are still the ones marked as most difficult to use.

However, individuals, persons with disability (40 respondents) and NGOs (48 respondents) in a targeted survey rated overall accessibility levels of online public services slightly more insufficient at the regional (30% individuals, 42% NGOs) and local level (27,5% individuals, 46% NGOs) than on national level (25% individuals, 31% NGOs).

3.1.2User experience

User experience was investigated with the same sub-sample of 198 participants (citizens, members of organisations representing persons with disabilities or older people, web accessibility experts or others) in the public consultation. A large majority affirms that their use of online public service increased slightly (42%) or significantly (37%) in the last three years.

Regarding ease of use, even though more than 57,7% of individuals that replied to the easy-to-read survey (379 out of 656) reported that government websites have become easier to use in the last three years; but when we look at the percentages of individuals finding government websites more difficult to use, it is almost three times higher among respondents identifying themselves as persons with a disability or with a recognised disability or impairment (almost 12% of both categories) than among those declaring to be persons without a disability (only the 4% of this category).

3.1.3Feedback mechanism and complaints

The same sub-sample of 198 participants of the public consultation (i.e., citizens, NGOs, accessibility experts and others) revealed a significant level of awareness about the feedback mechanism (73%) and the possibility to move a complaint to an enforcement body (71%).

However, a significant difference emerges among the different stakeholder groups. Awareness is very high among web accessibility experts (around 90% of them are aware of both feedback and complaints) and members of organisations representing persons with disabilities or older people (more than 70%) but is considerably lower among citizens (around 45% not aware). Similarly, in the easy-to-read survey, only 39% of the participants (255 out of 656) know how to complain to the government when an accessibility problem is not fixed.

Moreover, the feedback mechanism is still not widely used when accessibility issues arise. 44% of respondents (86 out of 198) have never given feedback to a public sector body. Among them, 46 respondents added a comment in the public consultation, which shows a high engagement rate. The three more common reasons were (i) users believe feedback do not have an effect, (ii) users did not know how to give feedback, and (iii) users did not have time to do so. This evidence is reinforced by the results of the easy-to-read survey, where 70% of individuals (457 out of 656) have never used the feedback mechanism. These results are confirmed by the findings of the in-depth interviews with the monitoring bodies, stating that the feedback mechanism was considered useful, but still not much used. One of the reasons provided was the low level of awareness among end users, which may be linked to the availability of accessibility statements, as many PSBs failed to add the statement which contains the information on how to provide feedback. Furthermore, among those respondents who used the feedback, the response received by the PSB was regarded in most cases as only partially satisfying (64%), and few respondents claim to be satisfied from the reply (13%). Similarly, 60% of individuals from the easy-to-read survey claim not to be happy with the answer to their feedback.

Similarly, the right to launch complaints has not yet been widely used. In the standard survey, 73% of respondents (145 out of 198) never complained to an ombudsman, a monitoring body or a human-rights network in their country. Among the reasons reported were the perceived uselessness of the mechanism and the inaccessibility of the complaint forms due to captcha. The percentage of users who never complained gets to 88% (576 out of 656) in the easy-to-read survey. Furthermore, few individuals (11 out of 40) participating in the targeted survey have complained to an enforcement body. The experiences reported by those individuals were mostly negative, as enforcement bodies have not always given a satisfactory response and were considered not aware of the mechanism. In case of complaints made, results differ between the standard version and the easy-to-read version. In the former, 81% of respondents claimed to be satisfied or partially satisfied with the response received. In the latter, 64% of individuals (51 out of 80 who complained) were not happy of how the complaint was managed.

3.2Scope and standards of the Directive

Almost half of the respondents (49%) of the standard survey (151 out of 311) agreed that the Directive adequately covers online public services to ensure full participation of people with disabilities in digital society. Regarding subject and content types not covered by the Directive (multiple choice), the majority of respondents favoured removing the current exemptions and/or expanding the scope: full inclusion of universities (81%), schools (70%), NGOs (56%), online maps (57%), live videos (53%), extranets/intranets (51%) and third-party content (47%) in view of technological advances.

Respondents identified the following top five technological advancements as relevant for the Directive: ‘artificial intelligence driven assistive technologies’ (60%), ‘accessible authoring tools’ (57%), ‘artificial intelligence and machine learning applied to monitoring, testing tools’ (43%), ‘Internet of Things (IoT) for accessible products and services’ (42%), ‘biometrics for identification and security aspects’ (31%).

Concerning the Harmonised European Standard EN 301 549 (v2.1.2), many respondents (63%) consider the standard useful for making public sector websites accessible. This view drops to 48% for mobile applications, and 25% for cross-border service offering. Around one third of respondents (32%) agree that the standard still covers all relevant end-user groups, with half (51%) not sure or neutral on the matter, and 17% disagreeing with the statement.

3.3Evaluation of the Directive

Each consultation was structured around the five evaluation criteria. The analysis presented here reflects main results from the public consultation, the targeted surveys and the in-depth interviews according to each criterion.

3.3.1Effectiveness

Increasing web accessibility

Regarding the first objective of the Directive (i.e., to increase accessibility of websites and mobile applications of public sector bodies), the consultations provided a composite picture. 66% of respondents (207 out of 311) of the standard survey considered the Directive successful, at least to some extent, in making online public services more accessible. The figures are slightly different when considering the targeted surveys to specific stakeholder groups. According to 50% of NGOs (24 out of 48), the Directive has contributed, at least to some extent, to increasing the accessibility. Similarly, 50% of individuals (20 out of 40) believed that the level of accessibility of digital public services improved since the implementation of the Directive.

In addition, the consultations also provided evidence on the availability and the perceived accessibility improvements on different online channels. Despite online information provided by PSBs has increased in the last three year according to 85% (265 out of 311) of participants (84% for interactive services, 78% for mobile applications, and 72% for online documents), when asked whether such online channels were made more accessible, the percentages were substantially lower (61% for online information, 43% for interactive services, 33% for mobile applications and 39% for online documents). The results of the targeted surveys with specific stakeholder categories are aligned with the results of the public consultation, with some differences in the magnitudes. Most of the PSBs (84%) reported a positive impact of the Directive on websites’ accessibility and, to a lower degree, on online documents (56%) and mobile applications (26%). Similarly, 71% of the NGOs reported a positive impact on websites’ accessibility and, to a lower degree, on online information (64%), interactive services (58%) and mobile applications (51%). Individuals provided slightly more negative answers. 51% of the respondents reported that there has been an improvement in accessibility of online information on websites, while significantly less individuals reported the same for interactive services (29%), mobile applications (20%) and online documents (19%). In addition, the answers to the targeted survey for all stakeholder categories suggest that the Directive has been mostly effective at improving accessibility at the national level, and, to a lesser extent, at the regional and local level.

The in-depth interviews conducted provided additional information on the achievement of the Directive’s first objective. Monitoring bodies reported that since the implementation of the directive they noticed a significant increase in the awareness of web accessibility among PSBs, which with time may lead to increased levels of accessibility in different online channels. In particular, the testing activities during the monitoring have been an important occasion of awareness raising.

Improving internal market

Moving to the second objective of the Directive (i.e., to harmonise the internal market for the accessibility of websites and mobile applications of public sector bodies), the most interesting results came from the in-depth interviews and the survey targeting technology providers. The other respondents to the public consultation reported high levels of uncertainty and the results from the survey targeting public sector bodies were inconclusive. While 30% of PSBs (19 out of 62) believed that the directive has increased cross-border provision of products and services for web-accessibility, 47% did not have an opinion on the topic. Moreover, 42% of the PSBs did not believe that the Directive has been effective at reducing prices of accessibility services. In line with this, 42 out of 311 participants (13.5%) of the public consultation considered reduced prices for accessibility tools and solution the least important key success factor for WAD.

The survey targeting industry players and technology providers aimed at exploring more in detail the issues related to the second objective of the Directive. 61% of participants (32 out of 52) believed that the Directive has increased cross-border demand of web accessibility products and services. This was beneficial for SMEs, according to 59% of them (31 out of 52). On the supply side, the impact on local companies has been reported to be slightly more significant than that on European companies in other countries (61% versus 56%). Moreover, 64% believed that the Directive also contributed to the development of new accessibility tools, and 59% reported an increase in the number of industry actors specializing in accessibility tools. However, only 22% reported an impact on the prices of digital accessibility services. In the in-depth interviews, there was little evidence for market changes. Rather the monitoring agencies reported lack of expertise as a major barrier for the success of WAD.

The results of the targeted survey were confirmed by the in-depth interviews with technology providers, who reported that the Directive has lowered barriers to entry other European markets, facilitating cross-border sales. Technology providers highlighted that tech companies are making considerable investments to provide better accessibility services and tools and that the Directive has created several new business opportunities for companies of all sizes.

Success factors and challenges

According to the participants of the public consultation, the three key success factors in the implementation of the Directive were (i) the accessibility statements (121 out of 311), (ii) the harmonisation of minimum accessibility requirements (118 out of 311) and (iii) the motivation, awareness and expertise of managers working in PSBs (113 out of 311).

Moreover, the 126 participants that replied that have been involved in the implementation of the Directive (including legislators, monitoring or enforcement bodies, a standardisation body, DPOs and PSBs), reported further opinions on the success factors. First, 76% of the participants of this sub-group (96 out of 126) believed that the Directive has contributed to the implementation of a formal monitoring procedure, while 60% believed the same for the formal enforcement procedure (76 out of 126). Finally, 57% believed that the minimum requirements introduced by the Directive have helped increase the availability of accessible websites and mobile applications (71 out of 126).

From the perspective of the monitoring bodies consulted during the in-depth interviews, one of the main success factors mentioned was the involvement of persons with disabilities and other stakeholders in the implementation of the Directive and the monitoring activities. However, the survey targeting NGOs provided a different picture. Most of the respondents reported little or no involvement of their organizations in the implementation of the Directive (60% - 29 out of 48) and the selection of websites and mobile applications to be monitored (64% - 31 out of 48). Despite the low degree of reported involvement, 67% of the respondents (33 out of 48) stated that the Directive helped organizations representing people with disabilities or older people in the EU to be heard on issues related to web accessibility.

Lastly, two other important success factors mentioned during the in-depth interviews with monitoring bodies were the facilitation of training and awareness raising in Member States and the frequent exchanges with the Web Accessibility Directive Expert Group (WADEX). However, according to only 35% of NGOs consulted in targeted surveys (17 out of 48), such initiatives have raised awareness, at least to some extent.

Moving to the challenges, the survey targeting PSBs (232 respondents) revealed that the main ones were related to (i) technical difficulties in making existing websites/apps compliant (38%), (ii) low level of accessibility knowledge in suppliers offering ICT products and services (33%), and (iii) lack of capacity of web managers working with accessibility at public sector bodies (32%). These results were confirmed by the in-depth interviews: several monitoring bodies (16) highlighted a lack of experts in the market, although an increase of competent market players is expected in the upcoming years. Furthermore, none of the monitoring bodies asked for a skill certificate when hiring external experts, mostly because often there are no official certification systems, which makes it more difficult to identify experts. Monitoring bodies would welcome for the next monitoring period clear and uniform instructions from the Commission to conduct the monitoring and to report the results.

3.3.2Efficiency

The consultation activities covered the efficiency evaluation criteria, gathering views and information on the costs and benefits related to the implementation of the Directive.

Costs

The public consultation did not provide conclusive responses related to the costs, as 58% of participants were not sure about whether the Directive’s objectives have been achieved at a reasonable cost. Some participants in the comments lamented the lack of available data to reply to this question. Only 18% agreed with the statement and 23% disagreed or strongly disagreed.

However, the consultations with specific stakeholders provided more information on the costs. 68% of the PSBs consulted (42 out of 62) with targeted survey believed that the Directive introduced additional economic burden to their organizations. Among them, (i) improving the accessibility of digital services, (ii) training of human resources and (iii) the creation of accessibility statements were the main sources of recurring costs. The Directive introduced some additional costs also to industry actors and technology providers, according to the responses to the targeted survey. In particular, 35% of them (18 out of 52) reported new costs associated to hiring new specialists, while 33% of them reported new investments to update products and services to meet the new accessibility requirements (17 out of 52). On the contrary, most of the (79%) reported no additional costs for their organisations (38 out of 48).

In addition, the Directive introduced new costs for the monitoring and enforcement bodies. According to the participants in the targeted survey, the main factor contributing to an increase in costs was represented by the burden of monitoring activities (30 out of 62). Monitoring bodies provided some information on this. However, the figures vary significantly and provide only a high-level overview and some examples, without the possibility to draw solid conclusions. The overall figures were reported only by few Member States and they range from €65,000 to €650,000 a year.

Benefits

The benefits brought by the implementation of the Directive were different depending on the stakeholder group. Overall, 59% of participants of the standard survey (184 out of 311) agreed that the benefits arising from the availability of accessible websites and mobile applications outweigh the costs of implementing the Directive. Some participants commented that accessibility is a human right, compared to which the costs are irrelevant. Looking at specific benefits, 56% of PSBs consulted appreciated the role of the Directive in inducing organizations to provide clearer accessibility requirements in procurement procedures. According to monitoring and enforcement bodies, the main benefits were (i) the harmonization of minimum requirements, (ii) the knowledge increase connected to the trainings, and (iii) the feedback, compliant and enforcement mechanisms, even if they are not yet frequently used. Finally, the industry players and technology providers reported during the interviews that the Directive brought benefits also for the private sector. The main benefits highlighted by the survey respondents were an increase in the demand of products and services in the national markets (27 out of 52) and an increase in the opportunities to penetrate different national markets (18 out of 52).

3.3.3Relevance

Most of the 311 respondents to the standard survey of the public consultation consider the specific objectives still relevant today, considering the evolving context of the last years. 97% confirm the relevance of providing accessible online content and services; 95% confirm that for mobile content and services, including mobile applications; 98% confirm that for digital inclusion of persons with disabilities and people with functional limitations; and 98% confirm that for reducing regulatory differences among Member States in the field of web accessibility.

While interviews with representatives of older people organisations confirmed that the objectives of the WAD are highly relevant also for older people, two specific issues emerged. First, a wide share of older people in Europe do not access public services online. Second, those who use them are not much aware of the provisions of the Directive (e.g., the feedback mechanism).

3.3.4Coherence

Most of the respondents to the standard survey of the public consultation believed that the Directive is coherent with existing EU legislation (54%), as well as with national legislations (56%). Similarly, 71% of the PSBs (44 out of 62) have never or rarely noticed any difference between the Directive and existing policy measures.

Monitoring bodies in the interviews reported that the Directive is coherent internally and with other key EU legislations, such as the European Accessibility Act. Finally, almost all monitoring bodies interviewed reported that there are no overlaps or inconsistency between the Directive and other relevant national measures related to web accessibility.

3.3.5EU added value

Most of the participants of the standard survey agreed that the Directive brought an added value in improving online accessibility for public services in their country (66% - 207 out of 311) and in increasing digital inclusion of users (53% - 165 out of 311).

Looking at the different stakeholder categories that replied to the targeted surveys, most of the PSBs (79% - 49 out of 62) and NGOs (72% - 35 out of 48) believed that the Directive contributed to making national laws on web accessibility requirements more harmonised. This opinion was also largely shared by monitoring bodies during the in-depth interviews. Similarly, the majority of PSBs (61% - 38 out of 62) believed that the Directive contributed to implementing the provisions of the UNCRPD and the EU Charter of Fundamental Rights at national level. According to industry players, the Directive brought an added value also on the internal market, by increasing the demand from public authorities (65% - 34 out of 52), as well as by increasing awareness about accessibility solutions among web professionals (62% - 32 out of 52).

Finally, the monitoring bodies interviewed reported that the main added value of the Directive has been the significant increase in the awareness of web accessibility among PSBs. In particular, the communication around monitoring results have been an important occasion of awareness raising.

4.List of interviewees

Table 3. List of interviewed stakeholders

Table 4. List of interviewed national monitoring, reporting and enforcement bodies

Annex VI. Additional considerations from national monitoring reports

Member State monitoring reports are published on national websites and made available also on the Commission’s website Web Accessibility Directive – Monitoring reports in original language and automated translations in English). According to these reports, total of 10 412 websites and 298 mobile applications have been tested for accessibility by MS during the first monitoring period, in what has been described as the ‘world’s largest accessibility test’.

The information provided in this Annex VI is complementary to findings and analysis in the main text of the evaluation report.

With regards to the administrative levels, the first monitoring reports show that almost all countries ensured a representative sample from local, regional and national websites of public sector bodies, taking as reference the Nomenclature of Territorial Units for Statistics  208 (NUTS) and the Local Administrative Units (LAU) set out in the NUTS. Not all NUTS and LAU levels exist in Latvia, Luxembourg and Malta. Therefore, reports from Latvia and Luxembourg  209 referred only to national and local public sector websites and not to any “regional” website. Malta also reported that it was not feasible for the Authority to consider websites at a regional level.

On sample size, the first monitoring reports show that most countries respected the minimum target  210 set in the Implementing Decision for all three monitoring methods: simplified monitoring of websites; in-depth monitoring of websites, and in-depth monitoring of mobile applications.

Table 5. Minimum sample size for monitoring

On the simplified monitoring of websites  212 , only Ireland and Finland declared a significantly lower-than-minimum number of websites monitored  213 (respectively less than 30% and less than 60%). Belgium and Malta significantly exceeded the minimum target (respectively monitoring 412 and 87 websites more than the minimum).

On the in-depth monitoring of websites  214 , eight countries did not reach the set minimum. Sweden monitored less than 80% of the minimum target sample, while Estonia monitored less than 75%, Greece less than 65%, and Ireland less than 30%. Some EU countries (Belgium, Germany, Malta and Poland) monitored in-depth a sample that was above the minimum target (these countries respectively monitored 6, 33, 14 and 75 websites more than the minimum). Latvia, Hungary, Romania and Slovenia did not report any data.

Finally, on mobile applications, two countries monitored a smaller than minimum sample (Ireland and Sweden), while Belgium, Latvia and Slovenia did not provide any data. Several EU Member States went beyond the minimum sample size: Bulgaria, Czechia, Germany, Estonia, Greece, Croatia, Lithuania, Hungary, Malta, Portugal, Romania and Slovakia.

The Implementing Decision laying down a monitoring methodology and the arrangements for reporting  215 specifies that the sample analysed in monitoring reports ‘shall include websites representing as much as possible the variety of services provided by the public sector bodies’. Three Member States did not report any breakdown of the sectors monitored. Most of the remaining 22 reports (from 16 countries) declared that the sample size covered all the sectors recommended in the Implementing Decision  216 , namely: social protection; health; transport; education; employment and taxes; environmental protection; recreation and culture; housing and community amenities; and public order and safety. In 11 countries, most websites monitored did not fall within the categories provided in the Implementing Decision and were included in a residual category of ‘other’. Among the categories provided, the three most monitored services were: (i) education; (ii) social protection; and (iii) recreation and culture. The two least monitored services were health and transport  217 .

For mobile applications, the Implementing Decision required different operating systems to be covered. 16 Member States reported data on how many Android-based or iOS-based applications were included in the sample. Four Member States monitored only one type of operating system application  218 , while two Member States did not report the distribution, and 3 (Belgium, Latvia and Slovenia) did not monitor mobile applications at all. Bulgaria reported that iOS-based applications were not used by a significant number of people with disabilities, because no adequate speech synthesiser is currently available in Bulgarian for that platform. Two countries provided a breakdown of the applications according to the service provided by public sector bodies, though this is not required by the Implementing Decision.

Annex VII. Mapping of UN Sustainable Development Goals (SDGs)

The mapping of SDGs in relation to WAD and its implementing acts was partly done using the SDG Mapper tool, developed by the JRC and DG INTPAJRC, https://knowsdgs.jrc.ec.europa.eu .

All of the UN Sustainable Development Goals (SDGs) are coherent with the achievement of more inclusion of people with disabilities in society  219 . The provisions of the WAD are, in particular, relevant for achieving the objectives of SDGs 1, 3, 4, 8, 9, 10, 16, and 17 as flagged by the UN Flagship Report on Disability and Sustainable Development Goals  220 .

Figure 2. SDG Goals and the percentage of corresponding keywords detected in the text of WAD and its implementing acts (calculated as the ratio of keywords in one Goal to the total number of keywords detected)

Achieving such goals requires the accessibility of public sector bodies’ digital administrative procedures, including services of general interest. The bullet points below describe how the key provisions of WAD and its implementing acts contribute to these SDGs  221 .

·The WAD aims at increasing the digital inclusion of people with disabilities and this is directly related to SDG 10 – ‘Reduce inequality within and among countries’. SDG 10 seeks to promote social, economic and political inclusion of all people (including people with disabilities) ensuring equal opportunities and ending discrimination (through Targets 10.2 and 10.3).

·The aim of increasing digital inclusion is also connected to SDG 16 – ‘Promote just, peaceful and inclusive societies’. SDG 16 seeks to ensure responsive, inclusive, participatory and representative decision-making at all levels (Target 16.7) as well as to promote and enforce non-discriminatory laws and policies for sustainable development (Target 16.b).

·Ensuring accessible communication also supports SDG 17 – ‘Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development’. Target 17.8 of this goal ‘aims to operationalize the Technology Bank and Science, Technology and Innovation capacity-building mechanism and enhance the use of enabling technology, in particular information and communications technology’.

·By aiming to increase the inclusion of online public services, the WAD can bring indirect positive impacts to the achievement of the following (sectoral) SDGs:

oSDG 1 – ‘Ending poverty in all its forms everywhere’;

oSDG 3 – ‘Ensure healthy lives and promote well-being for all at all ages’;

oSDG 4 – ‘Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all’;

oSDG 8 – ‘Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all’;

oSDG 9 – ‘Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation’, because application of the WAD contributes for technical developments of digital products and services to make them more accessible and inclusive (particularly target 9.c ‘significantly increase access to information and communications technology’).

Annex VIII. Bibliography

Angélico, M. J., Silva, A., Teixeira, S. F., Maia, T., & Silva, A. M. (2017). Web Accessibility and Transparency for Accountability: The Portuguese Official Municipal Websites. Research Paradigms and Contemporary Perspectives on Human-Technology Interaction, 140-166.

Alcántara, Ann-Marie, The Wall Street Journal (WSJ), More Companies Are Looking to Hire Accessibility Specialists, https://www.wsj.com/articles/more-companies-are-looking-to-hire-accessibility-specialists-11630501200, 1.9.2021

Barricelli, B. R., Sciarelli, P., Valtolina, S., & Rizzi, A. (2018). Web accessibility legislation in Italy: a survey 10 years after the Stanca Act. . Universal Access in the Information Society, 211-222.

Berlin Declaration on Digital Society and Value-based Digital Government (2020).

Berlin Declaration, Report on the monitoring of the Berlin Declaration, Directorate General for Informatics, (2022).

Berton, R., Gaggi, O., Kolasinska, A., Palazzi, C. E., & Quadrio, G. (2020). Are captchas preventing robotic intrusion or accessibility for impaired users. 2020 IEEE 17th Annual Consumer Communications & Networking Conference (CCNC), 1-6.

CAPGEMINI & DIALOGIC. (2017). The evaluation of implementation costs and future effects of the Web Accessibility Directive. Bruxelles: European Commission.

Easton, C. (2013). Website accessibility and the European Union: Citizenship, procurement and the proposed Accessibility Act. Int Rev Law Comput Technol, 187-199.

EDF. (2017). Toolkit - Directive on accessibility of the websites and mobile applications of public sector bodies. Bruxelles: Publication office of the EU.

Empirica, Funka, WRC. (2014). Measures to improve accessibility of public websites in Europe.

Eurofound. (2011). Measures for social inclusion of the elderly: The case of volunteering.

European Commission (2007). Measuring progress of eAccessibility in Europe – Assessment of the Status of eAccessibility in Europe, Main report.

European Commission (2010). Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – European Disability Strategy 2010-2020: A Renewed Commitment to a Barrier-Free Europe, COM(2010) 636 final.

European Commission (2011). Monitoring e-accessibility in Europe: 2010-2011, Annual report.

European Commission (2012). Impact assessment accompanying the document ‘Proposal for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies’ websites, SWD(2012) 401 final; Executive summary, SWD(2012) 402 final; Proposal for a Directive, COM(2012) 721 final.

European Commission (2013). G3ict, Study on Assessing and Promoting e-Accessibility, Final report.

European Commission (2017). Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – European Interoperability Framework – Implementation Strategy, COM(2017) 134 final.

European Commission. (2017). Quality of Public Administration. A Toolbox for Practitioners.

European Commission. (2018). e-Govenrment in Spain. European Commission.

European Commission (2018). EU-funded research projects into technologies for accessibility. Luxemburg: Publication office of the EU.

European Commission (2019). Shaping Europe’s digital future, Accessibility: Essential for Some, Useful for All, Factsheet/Infographic.

European Commission (2019). Peer Review on ‘Strategies for supporting social inclusion at older age’.

European Commission (2019). Survey on Web Accessibility Directive transposition and implementation. Bruxelles: Publication office of the EU.

European Commission (2020). Digital Education Action Plan 2021-2027, Resetting education and training for the digital age, COM(2020) 624 final.

European Commission (2020). Governance and public administration, Directorate-General for Structural Reform Support.

European Commission (2020). Communication ‘Identifying and addressing barriers to the Single Market’, COM(2020) 93 final.

European Commission (2021). Better regulation: guidelines and toolbox.

European Commission (2021). Joint Research Centre, Science Policy Briefs: Telework in the EU before and after the COVID-19: where we were, where we head to.

European Commission (2021), Joint Research Centre, Digitranscope: The governance of digitally-transformed society.

European Commission (2021). Staff Working Document – Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the Future, Commission Staff Working Document SWD(2021)101 final.

European Commission (2021). State-of-play report on digital public administration and interoperability, 2021, Directorate General for Informatics.

European Commission (2021). Union of Equality: Strategy for the Rights of Persons with Disabilities 2021-2030, COM(2021) 101 final.

European Commission (2022). Digital Skills and Jobs Platform.

European Commission (2022). eGovernment Benchmark 2022 & Background Report. Synchronising Digital Governments.

European Commission (2022). Joint Research Centre EU Science Hub, The Digital Competence Framework and DigComp 2.2.

European Commission (2022). Report from the Commission to the European Parliament and the Council on the implementation of the Recovery and Resilience Facility, COM(2022) 75 final.

European Commission (2022). Report from the Commission to the European Parliament and the Council – Review report on the implementation of the Recovery and Resilience Facility, COM(2022) 383 final.

European Committee of the Regions (2019). Resolution ‘the Green Deal in partnership with local and regional authorities’.

European Disability Forum (2019), Survey on Web Accessibility Directive transposition and implementation, presented at the 6th WADEX meeting, Brussels, 28 November 2019.

European Parliament (2020), European Parliamentary Research Service (EPRS), Employment and disability in the European Union, Briefing.

European Union (2012). Charter of Fundamental Rights of the European Union. Official Journal of the European Union.

Fiers, Wilco (2022). EU Runs World’s Largest Accessibility Test, www.deque.com/blog/eu-runs-worlds-largest-accessibility-test.

Ferri, D., & Favalli, S. (2018). Web Accessibility for People with Disabilities in the European Union: Paving the Road to Social Inclusion. Societies.

FRA. (2020). What do fundamental rights mean for people in the EU? Luxembourg: Publications Office of the European Union.

Freire, A. P., Russo, C. M., & Fortes, R. P. (2008). The perception of accessibility in web development by academy, industry and government: A survey of the Brazilian scenario. New Review in Hypermedia and Multimedia, 149-175.

Gambino, O., Pirrone, R., & DiGiorgio, F. (2016). Accessibility of the Italian institutional web pages: a survey on the compliance of the Italian public administration web pages to the Stanca Act and its 22 technical requirements for web accessibility. Univ Access Inf Soc, 305-312.

Grammenos, S. (2018). European Comparative Data On Europe 2020 & People With Disabilities. Centre de politique sociale et économique européenne Asbl (CESEP).

Hermann, M. (2018). What is Directive EU 2016/2102? Retrieved from https://medium.com/@marcusherrmann/what-isdirective-eu-2016-2102-a2ce364c77dd

Inal, Y., & Ismailova, R. (2020). Effect of human development level of countries on the web accessibility and quality in use of their municipality websites. Journal of Ambient Intelligence and Humanized Computing.

Loiacono, E. T., & McCoy, S. (2006). Website accessibility: a cross-sector comparison. Univ Access Inf Soc , 393-399.

LoPresti, E. F., Bodine, C., & Lewis, C. (2008). Assistive Technology for Cognition - Understanding the Needs of Persons with Disabilities. Quality of Life Technology.

Medeiros-Eler, M. J. (2018). Automated accessibility testing of mobile apps. IEEE 11th International Conference on Software Testing, Verification and Validation (ICST).

Ministru Kabineta. (2020). Kārtība, kādā iestādes ievieto informāciju internetā. Riga: Ministru Kabineta.

Parmanto, B. (2005). Metric for web accessibility evaluation. Journal of the American Society for Information Science and Technology.

Pribeanu, C. (2019). Large-scale accessibility evaluation of Romanian municipal websites. International Journal of User-System Interaction , 83-98.

Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019). Preliminary review of jobs, skills and competencies for implementation of Digital Accessibility. Proceedings of the Central European Conference on Information and Intelligent Systems.

Ringlaben, R., Bray, M., & Packard, A. (2014). Accessibility of American University Special Education Departments' Web sites. Universal Access in the Information Society.

Rubacek, F., Jindřichovská, I., Horváthová, Z., & Abrhám, J. (2020). Accessibility of Websites of the European National Tourism Boards. International Journal of Economics and Business Administration, 114-125.

Rubano, V., & Vitali, F. (2021). Making accessibility accessible: strategy and tools. 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). Las vegas: IEEE.

Sabev, N., Georgieva-Tsaneva, G., & Bogdanova, G. (2020). Research, analysis and evaluation of web accesibility for a selected group of public websites in Bulgaria. Journal of Accessibility and Design for All.

Sala, A., Arrue, M., Perez, J. E., & Valencia, X. (2019). Accessibility-in-use of public e-services: An exploratory study including users with low vision. INTERACCION’19.

Sanchez-Labella-Martin, I., Simelio, N., & Moreno-Sarda, A. (2017). Web accessibility for people with disabilities in Spanish city councils. Cuadernos, 155-172.

Sik-Lanyi, C., & Orban-Mihalyko, E. (2019). Accessibility Testing of European Health-Related Websites. Arabian Journal for Science and Engineering, 9171-9190.

Teixeira, L., Eusébio, C., & Silveiro, A. (2019, June). Website accessibility of Portuguese travel agents. In 2019 14th Iberian Conference on Information Systems and Technologies (CISTI) (pp. 1-6). IEEE.

Textkernel.nl, Remote Work, The current remote work trends in EMEA: An analysis across 8 European countries – Pre and during pandemic, 2018 – 2021.

Traxler, C. (2000). The Stanford Achievement Test, 9th Edition: National Norming and Perfirmance Standards for Deaf and Hard-of-Hearing students. Journal of Deaf Studies and Deaf Education 5.

UNDP. (2016). Human Development Report 2016. UNDP.

United Nations. (2006). Convention on the Rights of Persons with Disabilities. United Nations.

United Nations. (2014). General comment no. 2 (2014), Article 9, Accessibility: Committee on the rights of persons with disabilities. Geneva.

United Nations. (2015). Concluding observations on the initial report of the European Union, Committee on the Rights of Persons with Disabilities. Geneva.

United Nations. (2017). General comment No. 5 (2017) on living independently and being included in the community: Committee on the rights of persons with disabilities. Geneva.

United Nations. (2018). General comment no. 6 (2018) on equality and non-discrimination: Committee on the rights of persons with disabilities. Geneva.

Vasquez-Rodriguez, L. M. (2021). Investigating text simplification Evaluation. Cornell University.

Velleman, E. M., Nahuis, I., & VanDerGeest, T. (2017). Factors explaining adoption and implementation processes for web accessibility standards within eGovernment systems and organizations. Universal Access in the Information Society, 173-190.

Vollenwyder, B., Iten, G. H., Bruhlmann, F., Opwis, K., & Mekler, E. D. (2019). Salient beliefs influencing the intention to consider Web Accessibility. Computers in Human Behavior, 352-360.

W3C. (1999). Web Content Accessibility Guidelines 1.0.

W3C. (2008). Web Content Accessibility Guidelines 2.0.

The WebAIM Million. The 2022 report on the accessibility of the top 1,000,000 home pages.

Waddington, L. (2019). Regulating e-Accessibility and Digital Equality in Europe from a Multilevel Perspective. Scientifica, 3-18.

WHO. (2011). World Report on Disability. Geneva: WHO Press.

Yesilada, Y., Brajnik, G., Vigo, M., & Harper, S. (2014). Exploring perceptions of web accessibility: A survey approach. Behaviour & Information Technology, 119-134.

(1)     Directive (EU) 2016/2102 (OJ L 327, 2.12.2016). See also legislative summary of the Directive.

(2)    Implementing Decision (EU) 2018/1523 (OJ L 256, 12.10.2018) establishing a model accessibility statement; Implementing Decision (EU) 2018/1524 (OJ L 256, 12.10.2018) establishing a monitoring methodology and arrangements for reporting; and Implementing Decision (EU) 2018/2048 (OJ L 327, 21.12.2018) establishing a Harmonised European standard.

(3)    Petri H. et al. (2015): Towards a unified definition of web accessibility . W4A '15: Proceedings of the 12th International Web for All Conference. May 2015. Article No: 35, pp. 1-13.

(4)    Recital 2 of the WAD, cited above (note 1).

(5)    Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – Union of Equality: Strategy for the Rights of Persons with Disabilities 2021-2030, COM(2021) 101 final , 3.3.2021. See also its Monitoring framework .

(6)    European Commission (2019). Shaping Europe’s digital future, Accessibility: Essential for Some, Useful for All , Factsheet/Infographic (PDF) , 13 May 2019.

(7)    Commission Staff Working Document – Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the Future, SWD(2021)101 final , 9.4.2021.

(8)    Ferri, D., & Favalli, S. (2018). Web Accessibility for People with Disabilities in the European Union: Paving the Road to Social Inclusion. Societies, 8(2), 40.

(9)    Commission Staff Working Document – Impact Assessment, Accompanying the document ‘Proposal for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies’ websites’, SWD(2012) 401 final (PDF) ; Executive summary of the Impact Assessment, SWD(2012) 402 final (PDF) ; Proposal for a Directive, COM(2012) 721 final (PDF) .

(10)    According to the 2007 benchmarking study ‘Measuring progress of eAccessibility in Europe – Assessment of the Status of eAccessibility in Europe’ (MeAC 1), Main report (PDF) , p. 63, only 5,3% of government websites passed automatic and manual accessibility checks. In 2011, the follow-up study ‘Monitoring e-accessibility in Europe: 2010-2011’ (MeAC 2), Annual report (PDF) , p. 115, using a different method, found that 34% of the EU ‘web content’ (public and private) was accessible.

(11)    Impact Assessment, cited above (note 9), p. 25. It should be noted the original Proposal for the Directive, COM(2012) 721 final (PDF) , p. 19, covered only 12 types of public sector bodies’ websites, and did not cover mobile applications.

(12)    We refer here to what is officially reported by respondents in household surveys. It should be noted that the actual number of people with disabilities can vary between different data sources.

(13)    Grammenos, S., 2018. European Comparative Data On Europe 2020 & People With Disabilities. Centre de politique sociale et économique européenne Asbl (CESEP).

(14)    Eurostat (2020), (update planned in December 2022). See also overview of Disability statistics .

(15)    As provided in Article 288 of the TFEU . See also legislative summary about EU directives .

(16)    European Commission (2021), Better regulation: guidelines and toolbox .

(17)    Study supporting the review of the application of the Web Accessibility Directive (WAD), Final Report, PwC, Intellera Consulting, Open Evidence, Funka (2022).

(18)    See Annex I for procedural information and Annex II for details on the methodology.

(19)    Member State monitoring reports are published on national websites and made available also on the Commission’s website Web Accessibility Directive – Monitoring reports . See Annex V for stakeholder consultations and synopsis report.

(20)    80% of people with disabilities responding to public consultation replied to easy-to-read questionnaire (Annex V Synopsis report).

(21)    Source: Study, cited above (note 17). The relationship of the five key evaluation criteria and intervention logic is simplified for illustration purposes

(22)    Impact Assessment, cited above (note 9), p. 13.

(23)    European Commission (2007). Measuring progress of eAccessibility in Europe – Assessment of the Status of eAccessibility in Europe, Main report (PDF) .

(24)    European Commission (2011). Monitoring e-accessibility in Europe: 2010-2011, Annual report (PDF) .

(25)    European Commission (2013). G3ict, Study on Assessing and Promoting e-Accessibility, Final report . The web accessibility monitoring methodology used by Member States in recent reporting is different from MeAC studies and therefore the statistical results are not easily directly comparable (see below section 4.1).

(26)    Recital 9 and Article 1 of the WAD, cited above (note 1).

(27)    Recital 8 and Article 1 of the WAD, cited above (note 1).

(28)    The harmonised standard was updated by Commission Implementing Decision (EU) 2021/1339 of 11 August 2021 amending Implementing Decision (EU) 2018/2048 as regards the harmonised standard for websites and mobile applications (OJ L 289, 12.8.2021). The new harmonised standard is applicable since 12 February 2022, therefore it is not observed in this document.

(29)    Commission Implementing Decision (EU) 2018/1523 of 11 October 2018 establishing a model accessibility statement in accordance with Directive (EU) 2016/2102 of the European Parliament and of the Council on the accessibility of the websites and mobile applications of public sector bodies (OJ L 256, 12.10.2018).

(30)    Commission Implementing Decision (EU) 2018/1524 of 11 October 2018 establishing a monitoring methodology and the arrangements for reporting by Member States in accordance with Directive (EU) 2016/2102 of the European Parliament and of the Council on the accessibility of the websites and mobile applications of public sector bodies (OJ L 256, 12.10.2018).

(31)     Register of Commission expert groups and other similar entities (E03475) . See also its activities on Web Accessibility Directive Expert Group (WADEX) .

(32)    Meeting minutes available in Register of Commission expert groups and other similar entities (E03475) . Since 2016, the expert group met in total 32 times. From 2020 annual in-person meetings were replaced by series of online meetings and webinars. WADEX members collaborate on Futurium platform and Microsoft Teams.

(33)    Cited above (note 29).

(34)    Cited above (note 30).

(35)    Commission Implementing Decision (EU) 2018/2048 of 20 December 2018 on the harmonised standard for websites and mobile applications drafted in support of Directive (EU) 2016/2102 of the European Parliament and of the Council (OJ L 327, 21.12.2018).

(36)    Commission Implementing Decision (EU) 2021/1339 , cited above (note 28).

(37)     EN 301 549 V3.2.1 (2021-03) (PDF) , Harmonised European Standard, ‘Accessibility requirements for ICT products and services’, ETSI, CEN, CENELEC.

(38)     EUR-Lex portal: list of national transposition measures communicated by the Member States concerning Directive (EU) 2016/2102.

(39)    Except for the first monitoring period, which was two years for websites (1 January 2020 and 22 December 2021) and six months for mobile applications (23 June 2021 and 22 December 2021).

(40)    According to Recital 39 of the WAD, disproportionate burden means that the necessary measures “would impose an excessive organisational or financial burden on a public sector body, or would jeopardise the body's capacity to either fulfil its purpose or to publish information needed for or relevant to its tasks and services, while taking into account the likely resulting benefit or detriment for citizens, in particular persons with disabilities”.

(41)     List of Member States’ bodies in charge of monitoring, reporting and enforcement under the Directive.

(42)    This reflects the situation in September 2022, and all Member States that have not yet submitted reports have committed to do so. Although the WAD applies to EEA countries, transposition in these countries is not yet completed, and the EEA countries have not yet had an obligation to report.

(43)     List of Member States’ Monitoring reports (in original language and automated translations in English).

(44)    Wilco Fiers (Deque.com), EU Runs World’s Largest Accessibility Test , 1.02.2022.

(45)    2030 digital compass: the European way for the Digital Decade, COM(2021) 118 final , 9.3.2021. See also Europe’s Digital Decade: digital targets for 2030 .

(46)     eGovernment Benchmark (2022 ) , Insight Report (PDF) , p. 9. The benchmark compares how governments across Europe deliver digital public services. The method update of 2020 has led to a break in the series, which makes comparisons with earlier reports impossible.

(47)    Ibid., p. 18.

(48)     eGovernment Benchmark (2022), . The pilot tested 8 out of 50 Web Content Accessibility Guidelines (WCAG) 2.1 success criteria. A ‘pass’ means that content element is present on the web page (e.g. image) and it passes alternative text check. ‘Fail’ means it did not pass the check. ‘Inapplicable’ means that there is no image to test on the web page. In the sample, the total of all checks 76% passed, 17% failed, 7% were inapplicable. 83% websites passed five or more criteria.

(49)    eGovernment Benchmark (2022), Background Report (PDF) – Synchronising Digital Governments, pp. 20-22. Web accessibility foundations indicator pilot means that it is not included in the scoring of the current benchmark. The compliance rate of websites with the selected criteria ranged between 40,9% (WCAG: 1.4.3 colour contrast) and 99,5% (WCAG: 4.1.2 name, role, value), suggesting that some accessibility criteria are better implemented than others.

(50)    Cited above (notes 23-25). For example, 2010 MeAC 2 study, cited above (note 24), p. 118, evaluated conformity with WCAG 1.0/2.0, as WCAG 2.1 was only released in 2018 and reflected in EN 301 549 V2.1.2 (2018-08).

(51)     Berlin Declaration on Digital Society and Value-based Digital Government (2020) .

(52)    Report on the monitoring of the Berlin Declaration ( PDF ), Directorate General for Informatics, (2022), pp. 92-94.

(53)    MS policy action 2.2 “Ensure that the digital transformation is inclusive of and accessible for persons with disabilities and elderly persons and increase our efforts to make public services and information fully digitally accessible in accordance with the Web Accessibility Directive and the European Accessibility Act” and KPI 9 refer.

(54)    Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – European Interoperability Framework – Implementation Strategy, COM(2017) 134 final , 23.3.2017. See also The European Interoperability Framework .

(55)    Inclusion and accessibility (principle 7) measured through KPI-28 on compliance with the European accessibility standards – score improved from 2.25 (2019) to 2.61 (2021), EIF Monitoring dashboard . See also ‘State-of-play report on digital public administration and interoperability, 2021 ( PDF ), Directorate General for Informatics, pp. 23, 55. According to the report ‘20 different countries created or upgraded 29 public services in the period from 2020 to 2021’, p. 13.

(56)    Eurostat (2022), Code:  ISOC_CI_IFP_IU , Last internet use: in the last 12 months, all individuals. See also an overview of Digital economy and society indicators .

(57)    Eurostat (2022), Code: ISOC_CIEGI_AC , Internet use: interaction with public authorities, last 12 months; the share for all individuals was 58% (up from 44% in 2012).

(58)    Cited above (note 37), see also above, p. 9, regarding implementing decisions on harmonised standard.

(59)    Websites and mobile applications can be exempted from accessibility requirements only if those requirements impose a disproportionate burden on the public sector body. Disproportionate burden must be assessed before being applied, and must be explained in the relevant accessibility statement.

(60)    Other exceptions are listed in Article 1(4) of the Directive, for example, office file formats published before 2018, audio or video published before 2020, live audio or video, online mapping, third-party content, reproductions of heritage items, extranet and intranet content published before 2019, archives.

(61)    Cited above (note 30), Annex I, point 3(2)(c) states that ‘for the in-depth monitoring method the following pages and documents, if existing, shall be monitored: (c) the pages containing the accessibility statement or policy and the pages containing the feedback mechanism’.

(62)    See section 4.1.1 for a discussion on compliance with WAD.

(63)    Barricelli et al., 2018; Sala, Arrue, Perez & Valencia, 2019; Pribeanu, 2019; Sabev, Georgieva-Tsaneva & Bogdanova, 2020. The prevailing assessment method used in the literature was the use of automated checking tools, which has its limitations.

(64)    Cited above (notes 10, 23-25), see sections 1.1 and 2.1 of this report.

(65)    For example, 2010 MeAC 2 study, cited above (note 24), p. 118, evaluated conformity with WCAG 1.0/2.0, as WCAG 2.1 was yet to be released in 2018 and reflected in EN 301 549 V2.1.2 (2018-08).

(66)    See above section 3.2 in relation to eGovernment Benchmark (2022) Non-scored indicators (.xlsx) and notes 48, 49.

(67)    Ibid., WCAG 1.4.3 colour contrast criteria passed 40,9%, while WCAG 4.1.2 name, role, value criteria passed 99,5%.

(68)    The WebAIM Million . The 2022 report on the accessibility of the top 1 000 000 home pages.

(69)    Used by 80% of total respondents self-identifying as persons with disability.

(70)    See more in Annex V Synopsis report, section 3.1.1.

(71)    Denmark, Ireland, Finland, Sweden.

(72)    Greece, Spain, Italy, Cyprus, Malta, Portugal.

(73)    Belgium, Germany, France, Luxembourg, the Netherlands, Austria.

(74)    Bulgaria, Czechia, Estonia, Croatia, Latvia, Lithuania, Hungary, Poland, Romania, Slovenia, Slovakia.

(75)    As examples, the Netherlands have implemented a system of indicating compliance status in the accessibility statements (A: completely, B: partially, C: first measures taken, etc). Portugal offers usability and accessibility seal on websites (gold, silver, bronze) to showcase examples of compliance with best practices.

(76)     EN 301 549 V3.2.1 (2021-03) (PDF) , cited above (note 37), p. 181.

(77)    Implementing  Decision (EU) 2018/1524 establishing a monitoring methodology and the arrangements for reporting by Member States, cited above (note 30).

(78)    Ibid., Annex I, point 2.4.

(79)    Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019). Preliminary review of jobs, skills and competencies for implementation of Digital Accessibility. Proceedings of the Central European Conference on Information and Intelligent Systems.

(80)    Study, cited above (note 17), pp. 77, 81.

(81)     Automated Accessibility Testing Tools: How Much Do Scans Catch? (Essentialaccessibility.com, 2020).

(82)     Automated Testing Identifies 57 Percent of Digital Accessibility Issues (Deque.com, 2021).

(83)    Study, cited above (note 17), p. 49.

(84)     Directive (EU) 2019/882 of the European Parliament and of the Council of 17 April 2019 on the accessibility requirements for products and services (OJ L 151, 7.6.2019, p. 70-115).

(85)    In the public consultation , 30% of the PSB respondents (19 out of 62) agreed or strongly agreed that the Directive had increased the cross-border provision of products and services for web accessibility. Only 10% disagreed or strongly disagreed with the statement, and most of the remaining respondents (60%) neither agreed nor disagreed.

(86)    See Annex V Synopsis report, section 4 List of interviewees.

(87)    65% of industry respondents to the targeted survey (34 out of 52) reported that the provisions included in the WAD had fostered the development of new accessibility tools and products, at least to some extent, according to Study, cited above (note 17), p. 55.

(88)    Entering into practical effect in 2025 on a range of digital hardware and services.

(89)    See Annex VI of this report for details on sample sizes, and information on monitoring and reporting exercise.

(90)    See section 3.2, and national monitoring reports, cited above (notes 42, 43).

(91)    Study, cited above (note 17), p. 46.

(92)    According to the WCAG 2.1 guidelines, AAA success criteria relate to the highest level of accessibility. The HEN refers to A and AA level success criteria from the WCAG 2.1 guidelines.

(93)    Estonia, Finland, Greece, the Netherlands and Spain.

(94)    The minimum sample size is defined in the Decision (EU) 2018/1524 , reproduced in Annex IX of this report.

(95)    According to Implementing Decision (EU) 2018/1524 and in line with recommendations of UN Committee General comment No. 7 (2018) on the participation of persons with disabilities, including children with disabilities, through their representative organizations, in the implementation and monitoring of the Convention.

(96)    W3C, Accessibility Conformance Testing (ACT). For example, according to a blog post ‘ Harmonized Accessibility Testing ’ (30.07.2019) by Shadi Abou-Zahra, ‘a method may mistakenly fail text alternatives that are too long when that is not actually defined by WCAG. Another method may correctly mark that as a warning because it is advisory good practice.’

(97)    European Commission (2022), Web Accessibility Directive – Standards and harmonisation (HTML) : EU legislation, technical standards and W3C international best practice on web accessibility.

(98)    The legislator was aware of this challenge, therefore website and mobile application sample sizes are based on population of a Member State; Implementing Decision (EU) 2018/1524, cited above (note 30), Annex I, point 2.

(99)    Confirmed by 11 MS: Czechia, Denmark, Germany, Greece, Spain, Italy, Latvia, Malta, Netherlands, Austria, Romania, and Sweden.

(100)    Confirmed by 12 MS: Germany, Greece, Spain, Croatia, Latvia, Lithuania, Luxembourg, Hungary, Poland, Slovakia, Finland, Sweden.

(101)    Confirmed by 12 MS: Denmark, Germany, Estonia, Greece, Croatia, Italy, Luxembourg, Hungary, Malta, Slovakia, Finland, Sweden.

(102)    Velleman, E. M., Nahuis, I., & VanDerGeest, T. (2017). Factors explaining adoption and implementation processes for web accessibility standards within eGovernment systems and organizations. Universal Access in the Information Society, pp. 173-190.

(103)    Implementing Decision (EU) 2018/1524, cited above (note 30), Annex I, points 2.2.4 and 2.3.5: ‘Member States shall consult national stakeholders, in particular organisations representing persons with disabilities, on the composition of the sample of the websites to be monitored, and give due consideration to the stakeholders’ opinion regarding specific websites [and mobile applications] to be monitored’.

(104)    European Disability Forum (2019), Survey on Web Accessibility Directive transposition and implementation, presented at the 6th WADEX meeting, Brussels, 28 November 2019, (download Survey PPTX or see meeting documents on WADEX page ).

(105)    Commission Implementing Decision (EU) 2018/1524 , cited above (note 30).

(106)    France, Croatia, Malta, Poland, Romania, Slovenia and Slovakia; fines range from EUR 200 to EUR 25 000, depending on the Member State.

(107)    88% of respondents to the easy-to-read survey have never complained and 61% did not know how to complaint (Annex V, section 3.1.3 Feedback mechanism and complaints).

(108)    See above (note 106).

(109)    Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019), cited above (note 79). The authors highlight that training new professionals to the adequate level of competencies in the sector is extremely beneficial towards achieving the goals of the Directive, improving both the internal market of jobs, and the quality of life of end-users with disabilities.

(110)    For example, EU co-funded Erasmus+ Certified Digital Accessibility Training project offers specialised courses for: Digital Accessibility Manager, Digital Accessibility Tester, Web Developer with expertise in Digital Accessibility, and Web Designer with expertise in Digital Accessibility.

(111)    Rubano, V., & Vitali, F. (2021). Making accessibility accessible: strategy and tools. 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). Las Vegas: IEEE.

(112)    Alcántara, Ann-Marie, WSJ, More Companies Are Looking to Hire Accessibility Specialists , 1.9.2021.

(113)     https://ec.europa.eu/eures/portal/jv-se (search on 25 September 2022), for all European countries, with ‘accessibility’ keyword.

(114)    Study, cited above (note 17), p. 63.

(115)    Joint Research Centre EU Science Hub (2022), The Digital Competence Framework .

(116)    Ibid., DigComp 2.2 - The Digital Competence Framework for Citizens (PDF).

(117)    According to press statement by the executive Vice-President for A Europe Fit for the Digital Age, Margrethe Vestager: ‘The Digital Decade is about making digital technology work for people and businesses. It is about enabling everyone to have the skills to participate in the digital society. It is about bringing government services closer to citizens. Europe's digital transformation will give opportunities for everyone.’ 14.7.2022.

(118)    More than double compared to 2021 when almost 9 million people in the EU worked as ICT specialists, representing 4.5% of the total EU workforce.

(119)    Regulation (EU) 2021/694 of the European Parliament and of the Council of 29 April 2021 establishing the Digital Europe Programme and repealing Decision (EU) 2015/2240 (OJ L 166, 11.5.2021). The programme aims, among others, to bridge the digital divide, in particular under specific objective 4: advanced digital skills, with indicative envelope of EUR 577 million.

(120)    Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the Future, SWD(2021)101 final , cited above (note 7), p. 11.

(121)    European Commission (2021), Joint Research Centre, Science Policy Briefs: Telework in the EU before and after the COVID-19: where we were, where we head to (PDF) .

(122)    Textkernel.nl, Remote Work, The current remote work trends in EMEA : An analysis across 8 European countries – Pre and during pandemic, 2018 – 2021.

(123)    European Parliament (2020), European Parliamentary Research Service (EPRS), Employment and disability in the European Union, Briefing (PDF) , p. 2, “the unemployment rate of persons with disabilities in the EU, aged 20-64, is 17.1% compared to 10.2% of persons without disabilities”.

(124)    Study, cited above (note 17), pp. 62, 65.

(125)    International Association of Accessibility Professionals (IAAP), https://www.accessibilityassociation.org/s/certification .

(126)    2030 digital compass, COM(2021) 118 final , cited above (note 45).

(127)    Study, cited above (note 17) and Annex V Synopsis report. According to targeted survey for persons with disabilities or those taking care of persons with disabilities (n=40): respondents noticed improvements in accessibility of online information at national level (51%), regional level (45%) and local level (38%). According to another survey for organisations representing persons with disabilities, older people or consumers (n=48), respondents rated overall accessibility levels of public sector online services less insufficient on national level (31%) than on regional (42%) or local level (46%).

(128)    The Recovery and Resilience Facility is the key instrument at the heart of NextGenerationEU to help the EU emerge stronger and more resilient from the current crisis. Recovery and Resilience Scoreboard (2022): Digital transformation. Report from the Commission to the European Parliament and the Council – Review report on the implementation of the Recovery and Resilience Facility, COM(2022) 383 final (PDF), 29.7.2022. For thematic analysis and project examples, see Thematic analysis –Digital public services (PDF), 2021.

(129)    Sik-Lanyi, C., & Orban-Mihalyko, E. (2019). Accessibility Testing of European Health-Related Websites. Arabian Journal for Science and Engineering, 9171-9190; Teixeira, L., Eusébio, C., & Silveiro, A. (2019, June). Website accessibility of Portuguese travel agents. In 2019 14th Iberian Conference on Information Systems and Technologies (CISTI) (pp. 1-6). IEEE.

(130)    Study, cited above (note 17), p. 46.

(131)    Ibid., other examples in Finland are: suppliers of secure electronic identification services; credit institutions; payment institutions; investment firms.

(132)    Spain, France, Italy, the Netherlands, Poland, Romania, Slovakia, and various states and regions in Belgium, Germany and Austria.

(133)    The WAD does not apply to content of websites and mobile applications listed in its Article 1(4), e.g. live time-based media.

(134)    Study, cited above (note 17), ‘2.3 Technological advances in web accessibility’, pp. 27-29.

(135)    See Annex V, point 3.2.

(136)    Study, cited above (note 17), Annex F ‘Technological advancements in web accessibility’, p. 221-241. The 13 technologies reviewed are: speech recognition, biometrics and face recognition, automated accessibility testing, automated text simplification, brain-computer interface, virtual and augmented reality devices, sign language recognition, web authoring tools, automated summarisation, automated image annotation, automated lip reading, speaker recognition, wearable devices. Scientific fields covered are Natural Language Processing, Cloud computing, Neural networks and deep learning, Computer vision, Optical Character Recognition, Gesture recognition.

(137)     https://www.easyreading.eu/ .

(138)    Study, cited above (note 17), pp. 83-85. See also section 5.1.3 of this report for further analysis regarding relevance of these technological advancements.

(139)    E.g. ‘SignOn project’, https://signon-project.eu/ , and ‘EASIER’ https://www.project-easier.eu/ (EU’s Horizon 2020 programme).

(140)    For example, the Commission has made its language tools, including automated eTranslation and speech transcription services available to all European public sector bodies, SMEs, academia, and NGOs at https://language-tools.ec.europa.eu/ .

(141)    European Language Grid (EU’s Horizon 2020 programme), https://live.european-language-grid.eu/catalogue/dashboard (select all EU languages, and technological factors).

(142)    Study, cited above (note 17), p. 79.

(143)    The Directive and Implementing Decision (EU) 2018/1524, cited above (note 30), does not explicitly require reporting on costs.

(144)    Digital Skills and Jobs Platform, https://digital-skills-jobs.europa.eu/en .

(145)    Study, cited above (note 17), pp. 74, 80.

(146)    Here are only some examples and highlights provided. For a detailed overview of EU support measures, see Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the Future, SWD(2021)101 final , Section 4, cited above (note 7).

(147)    Report from the Commission to the European Parliament and the Council on the implementation of the Recovery and Resilience Facility, COM(2022) 75 final , 1.3.2022.

(148)     Recovery and Resilience Scoreboard (2022): Digital transformation, ;   , cited above (note 128).

(149)     DESI: Digital infrastructures 2022 (PDF) , p.9.

(150)    Regulation (EU) 2021/694 of the European Parliament and of the Council of 29 April 2021 establishing the Digital Europe Programme and repealing Decision (EU) 2015/2240 (OJ L 166, 11.5.2021). The programme aims, among others, to bridge the geographical digital divide, including the outermost regions, in particular under specific objective 4: advanced digital skills and specific objective 5: deployment and best use of digital capacity and interoperability.

(151)    European Commission, European Structural and Investment Funds. Percentage of cohesion policy funding in public investments per Member State 2015-2017 .

(152)     Regulation (EU) 2021/1060 of the European Parliament and of the Council of 24 June 2021 laying down common provisions on the European Regional Development Fund, the European Social Fund Plus, the Cohesion Fund, the Just Transition Fund and the European Maritime, Fisheries and Aquaculture Fund and financial rules for those and for the Asylum, Migration and Integration Fund, the Internal Security Fund and the Instrument for Financial Support for Border Management and Visa Policy (OJ L 231, 30.6.2021), in particular Articles 2(3) and 74(1)(a).

(153)    For example, according to Commission’s Kohesio database , digital accessibility of local government sites project in Poland, with a budget of 1,1 million euro and co-funded from European Social Fund, aims to train public administration officials to create digitally accessible resources and to develop IT tools to support the process of monitoring the implementation of digital accessibility in public administrations.

(154)    Quality of Public Administration. A Toolbox for Practitioners , 2017, in particular Theme 5: Service delivery and digitalisation (PDF) .

(155)     Technical Support Instrument (TSI).

(156)    European Commission, Directorate-General for Structural Reform Support, Governance and public administration (PDF) , 2020.

(157)     https://knowledge4policy.ec.europa.eu/home_en .

(158)     https://ai-watch.ec.europa.eu/topics/public-sector_en .

(159)     https://cordis.europa.eu/project/id/780057 (EU contribution EUR 1 999 812), project website https://www.w3.org/WAI/about/projects/wai-tools/ .

(160)     https://cordis.europa.eu/project/id/780206 (EU contribution EUR 1 957 011,25), project website https://wadcher.eu/ .

(161)     https://cordis.europa.eu/project/id/822245 (EU contribution EUR 1 499 742), project website https://www.w3.org/WAI/about/projects/wai-guide/ .

(162)    Pilot on web accessibility for web authoring tools producers and communities (budget EUR 150 000), https://www.funka.com/en/projekt/we4authors/what-is-we4authors .

(163)    European Disability Strategy 2010-2020: A Renewed Commitment to a Barrier-Free Europe, COM(2010) 636 final (PDF), 15.11.2010.

(164)    Strategy for the Rights of Persons with Disabilities 2021-2030, cited above (note 5).

(165)    Digital Education Action Plan 2021-2027, Resetting education and training for the digital age, COM(2020) 624 final (HTML) , 30.09.2020. Related content on https://education.ec.europa.eu/focus-topics/digital-education/action-plan .

(166)    European Commission (2020), Renovation Wave: doubling the renovation rate to cut emissions, boost recovery and reduce energy poverty (HTML) .

(167)    Cited above (note 84).

(168)     Regulation (EU) 2022/2065 of the European Parliament and of the Council of 19 October 2022 on a Single Market For Digital Services and amending Directive 2000/31/EC (Digital Services Act) (OJ L 277, 27.10.2022); Press release 16.11.2022, ‘Digital Services Act: EU’s landmark rules for online platforms enter into force’.

(169)     Directive 2010/13/EU of the European Parliament and of the Council of 10 March 2010 on the coordination of certain provisions laid down by law, regulation or administrative action in Member States concerning the provision of audiovisual media services (Audiovisual Media Services Directive) (OJ L 95, 15.4.2010).

(170)     Directive (EU) 2018/1972 of the European Parliament and of the Council of 11 December 2018 establishing the European Electronic Communications Code (Recast) (OJ L 321, 17.12.2018).

(171)     Directive (EU) 2017/1564 of the European Parliament and of the Council of 13 September 2017 on certain permitted uses of certain works and other subject matter protected by copyright and related rights for the benefit of persons who are blind, visually impaired or otherwise print-disabled and amending Directive 2001/29/EC on the harmonisation of certain aspects of copyright and related rights in the information society (OJ L 242, 20.9.2017).

(172)     Regulation (EU) 2017/1563 of the European Parliament and of the Council of 13 September 2017 on the cross-border exchange between the Union and third countries of accessible format copies of certain works and other subject matter protected by copyright and related rights for the benefit of persons who are blind, visually impaired or otherwise print-disabled (OJ L 242, 20.9.2017).

(173)     Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market and repealing Directive 1999/93/EC (OJ L 257, 28.8.2014).

(174)    The WAD does not apply to office file formats published before 23 September 2018, whereas in EAA that date is 28 June 2025.

(175)     Charter of Fundamental Rights of the European Union (OJ C 326, 26.10.2012).

(176)    European Pillar of Social Rights, https://ec.europa.eu/info/strategy/priorities-2019-2024/economy-works-people/jobs-growth-and-investment/european-pillar-social-rights_en .

(177)    UN Convention on the Rights of Persons with Disabilities (UNCRPD), https://www.un.org/development/desa/disabilities/convention-on-the-rights-of-persons-with-disabilities.html .

(178)    UN Sustainable Development Goals (SDGs), https://sdgs.un.org/goals .

(179)    European Declaration on Digital Rights and Principles for the Digital Decade, COM(2022) 28 final , 26.1.2022. See more at https://digital-strategy.ec.europa.eu/en/policies/digital-principles .

(180)    E.g., the definitions for communication, reasonable accommodation, universal design (Article 2); general principles of accessibility (Article 3(f)); general obligations (Article 4(a, b, c, h)) of the UNCRPD.

(181)    E.g., on Living independently and being included in the community (Article 19(a, b, c)) requiring that community services and facilities for the general population are available on an equal basis to persons with disabilities and are responsive to their needs; Work and employment (Article 27(a, c, e, h)); Adequate standard of living and social protection (Article 28(b, c, d, e); Participation in political and public life (Article 29); and on ensuring that international cooperation is inclusive of and accessible to persons with disabilities (Article 32(a)) of the UNCRPD.

(182)     https://www.un.org/development/desa/disabilities/publication-disability-sdgs.html .

(183)     https://social.un.org/publications/UN-Flagship-Report-Disability-Final.pdf .

(184)    The mapping of SDGs in relation to WAD was partly done using the SDG Mapper tool, developed by the JRC and DG INTPAJRC, https://knowsdgs.jrc.ec.europa.eu/ .

(185)    Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019), cited above (note 79). See also section 4.1.7.

(186)    Study, cited above (note 17), p. 106.

(187)    See Annex V, section 3.3.1.

(188)    In particular SDGs 1, 3, 4, 8, 9, 10, 15, 16 and 17 (see section 4.1.1 and Annex VII).

(189)    Strategy for the Rights of Persons with Disabilities 2021-2030, cited above (note 5), point 8.2, where the Commission commits to adopt ‘an action plan on web accessibility, to be shared and promoted in all EU institutions, bodies and agencies in view of ensuring compliance of EU websites, documents published on these websites and online platforms, with European accessibility standards’.

(190)    European Commission (2022), Web Accessibility Action Plan 2022–2025 (HTML); which sets out the steps the Commission must take if it is to achieve its stated ambitions and, in particular, ensure that its web presence is fully compliant with the provisions of the WAD.

(191)    This is in particular recognized in the new European Care Strategy and Commission Proposal for a Council Recommendation on access to affordable high-quality long-term care, COM(2022) 441 final (PDF) . It calls for rolling-out accessible innovative technology and digital solutions in the provision of care services, including to support independent living for persons with disability (point 5(c)).

(192)    Digital technologies affect the way in which every government function is carried out, from decision-making to delivering services, collecting and managing tax, communicating, managing projects, etc. See European Commission. Joint Research Centre (2021). Digitranscope: The governance of digitally-transformed society .

(193)    The two general objectives are to ensure websites and mobile applications of PSBs are more accessible and to harmonise the internal market for web accessibility products and services.

(194)    See, in particular, Implementing Decision (EU) 2018/1524, cited above (note 30), Annex I, point 1.3.2.

(195)    Ares(2020)4553282. ISG members: SG, SJ, COMM, DIGIT, GROW, EMPL, JRC, SANTE, JUST, ESTAT, CNECT.

(196)    Have your say portal, Published initiatives: Accessible web & digital content for people with disabilities – review of EU rules .

(197)    Disability coordinators representing COMP, ECFIN, EAC, HOME, REGIO, TAXUD. Disability coordinators have been nominated in all Commission DGs under the Strategy for the Rights of Persons with Disabilities 2021-2030, COM(2021) 101 final , cited above (note 5). See also its Monitoring framework .

(198)    See Annex II, section 6.

(199)     Register of Commission expert groups and other similar entities (E03475) , and see above (note 32) . See also its activities and meeting minutes on Web Accessibility Directive Expert Group (WADEX) .

(200)    Member State monitoring reports are published on national websites and made available also on the Commission’s website Web Accessibility Directive - Monitoring reports (4.01.2022).

(201)    See above section 3.2 and Annex II, section 4.

(202)     Web Accessibility Directive - Monitoring reports (4.01.2022).

(203)    Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies (OJ L 327, 2.12.2016).

(204)    Summary report of the public consultation, with survey data is published online, Have your say portal, Published initiatives: Accessible web & digital content for people with disabilities – review of EU rules/public consultation .

(205)    See list of interviewees in section 4 of this Annex below.

(206)     EU Survey tool .

(207)    It is important to note that responses in the public consultation as well as the targeted surveys represent more opinions from countries where web accessibility policies were in place before the adoption of WAD. This imbalance of the statistical data must be taken into consideration when analysing the results.

(208)     https://ec.europa.eu/eurostat/web/nuts/background .

(209)    Luxembourg stated that no website at regional level was found in the country’s inventory of public websites, source: https://data.public.lu/fr/datasets/inventaire-des-sites-publics/ .

(210)    Source: Study supporting the review of the application of the Web Accessibility Directive (WAD), Final Report, PwC, Intellera Consulting, Open Evidence, Funka, 2022, cited above (note 17). No information available yet for France and Cyprus.

(211)    The Implementing Decision (EU) 2018/1524 provides exceptions for Member States which have an overall number of websites or mobile applications lower than the size of the population-based sample. In such cases, MS shall monitor at least 75% of all websites available, and at least 50% of all mobile applications available.

(212)    Simplified monitoring is a method applied to websites that detects instances of non-compliance with a sub-set of the requirements in the standards.

(213)    It should be noted that these Member States have committed to update their national report in 2022.

(214)    In-depth monitoring method thoroughly verifies whether a website or mobile application satisfies all the requirements identified in the standards. Ideally this should be performed by users with a range of disabilities and on a variety of devices.

(215)    Annex I, section 2.2.3.

(216)    Ibid.

(217)    This statement is based on the figures provided by Member States specifying the number of websites monitored for each sector, namely all the Member States except for Ireland and Bulgaria.

(218)    Three Member States monitored only Android-based applications and 1 Member State monitored only iOS-based applications. Czechia stated that iOS-based applications could not be monitored due to the lack of available techniques.

(219)     https://www.un.org/development/desa/disabilities/publication-disability-sdgs.html .

(220)     https://social.un.org/publications/UN-Flagship-Report-Disability-Final.pdf .

(221)    The mapping of SDGs in relation to WAD was partly done using the SDG Mapper tool, developed by the JRC and DG INTPAJRC, https://knowsdgs.jrc.ec.europa.eu/ .