This document explores the page-based conformance verification approach used by WCAG 2.0 and 2.1 accessibility guidelines. It explains how this approach is challenging to apply to certain websites and web applications. It also explores ideas on how future versions of guidelines might address these challenges. This document focuses primarily on challenges to large, highly complex, dynamic sites. Other efforts in WAI are looking at different aspects of conformance for other types of sites.
- +The challenges covered broadly fall into five main areas:
Accessibility Supportedin the many provisions -tied to use with assistive technologies and platform accessibility -features, combined with the lack of definition of what constitutes -Accessibility Supported, further exacerbates the need for expert -human judgement (#1 above), as well as potential different and -non-overlapping sets of these features used when including 3rd -party content (#3 above).
Accessibility Supportedin the many provisions tied to use with assistive technologies and platform accessibility features, combined with the lack of definition of what constitutes Accessibility Supported, further exacerbates the need for expert human judgement (#1 above), as well as potential different and non-overlapping sets of these features used when including 3rd party content (#3 above).
The purpose of this document is to help understand those challenges more holistically, and explore approaches for mitigating them so that we can address such challenges more fully in future accessibility guidelines including the forthcoming W3C Accessibility Guidelines (WCAG 3.0) (now in early development) where the W3C Working Group Charter expressly anticipates a new conformance model.
@@ -199,47 +193,45 @@This document has two key goals:
-Silver,as well as through active discussion in the Silver Conformance Options Subgroup.
Silver,as well as through active discussion in the Silver Conformance Options Subgroup.
A better understanding of the situations in which the WCAG 2.x conformance model may be difficult to apply could lead to more effective conformance models and testing approaches in the future.
It is important to recognize that success criteria in WCAG 2.x are quite distinct from the conformance model. These criteria describe approaches to content accessibility that are thoughtfully designed to enable people with a broad range of disabilities to effectively consume and interact with web content. Challenges with the conformance model do not in any way invalidate the criteria. For example, while requiring human judgment to validate a page limits testing to sampling of templates, flows, and top tasks, etc. (see Challenge #1 below), without that human judgement it may not be possible to deliver a page that makes sense to someone with a disability. Similarly, while it may not be possible to know that all third party content is fully accessible (see Challenge #3 below), without review of that content by someone sufficiently versed in accessibility it may not be possible to be sure that pages containing third party content fully conform to WCAG 2.x. Human judgement is a core part of much of WCAG 2.x for good reasons, and the challenges that arise from it are important to successfully grapple with.
-This document is published to seek additional contributions from the wider web community on:
-We seek to gain a thorough understanding of the challenges faced by large, complex, and dynamic websites who are attempting to provide accessible services to their web site users. It is expected that a more thorough understanding of these challenges can lead to either a new conformance model, or an alternative model that is more appropriate for large, complex, and/or dynamic websites (in WCAG 3.0).
+This document is published to seek additional contributions from the wider web community on:
+We seek to gain a thorough understanding of the challenges faced by large, complex, and dynamic websites who are attempting to provide accessible services to their web site users. It is expected that a more thorough understanding of these challenges can lead to either a new conformance model, or an alternative model that is more appropriate for large, complex, and/or dynamic websites (in WCAG 3.0).
-This document also includes previously published research from the Silver Task Force and Community Group that is related to Challenges with Accessibility Guidelines Conformance and Testing. There is some - overlap between the challenges captured in this published research and the challenges enumerated in the first 4 sections of this document. The research findings have been folded into other sections of this document as appropriate. -
+This document also includes previously published research from the Silver Task Force and Community Group that is related to Challenges with Accessibility Guidelines Conformance and Testing. There is some overlap between the challenges captured in this published research and the challenges enumerated in the first 4 sections of this document. The research findings have been folded into other sections of this document as appropriate.
-Also present in this document is an introductory discussion of approaches to mitigate the impact of the challenges cited that have been suggested by various stakeholders. We are publishing this updated draft now to continue seeking wide review to further catalogue and characterize the challenges and mitigation approaches, so that this work can become input into W3C accessibility guidelines (WCAG 3.0). +
Also present in this document is an introductory discussion of approaches to mitigate the impact of the challenges cited that have been suggested by various stakeholders. We are publishing this updated draft now to continue seeking wide review to further catalogue and characterize the challenges and mitigation approaches, so that this work can become input into W3C accessibility guidelines (WCAG 3.0).
The following terms are used in this document:
-The following terms are used in this document:
+A challenge common to many success criteria is the inability for automatic testing to fully validate conformance and the subsequent time, cost, and expertise needed to perform the necessary manual test to cover the full range of the requirements. @@ -249,30 +241,13 @@
Appendix A describes challenges with applying the WCAG 2.x conformance model to specific Guidelines and Success Criteria, primarily based on required human involvement in evaluation of conformance to them. The list is not exhaustive, but it covers the preponderance of known challenges with all A and AA Success Criteria.
Silver research identified two further challenges related to scaling conformance verification:
- -What is Strictly Testable- Silver finds that
The requirement for valid and reliable testability for WCAG success criteria presents a structural barrier to including the needs of people with disabilities whose needs are not - strictly testable.User needs such as thos articulated by the W3C's Cognitive and Learning Disabilities (COGA) Task Force in their extensive W3C Note publication Making content usable for people with cognitive and learning disabilities [[coga-usable]] only expand and exacerbate the need for expert human testing. As silver also notes:
The entire principle of understandable is critical for people with cognitive disabilities, yet success criteria - intended to support the principle are not easy to test for or clear on how to measure.-
Regardless of proficiency, - there is a significant gap in how any two human - auditors will identify a success or fail of criteria. - … Ultimately, there is variance between: any two - auditors; … Because there's so much room for human - error, an individual may believe they've met a specific - conformance model when, in reality, that's not the - case. … There isn't a standardized approach to how - the conformance model applies to success criteria at - the organizational level and in specific test case - scenarios.
Silver research identified two further challenges related to scaling conformance verification:
+ +What is Strictly TestableSilver finds that
The requirement for valid and reliable testability for WCAG success criteria presents a structural barrier to including the needs of people with disabilities whose needs are not strictly testable.User needs such as thos articulated by the W3C's Cognitive and Learning Disabilities (COGA) Task Force in their extensive W3C Note publication Making content usable for people with cognitive and learning disabilities [[coga-usable]] only expand and exacerbate the need for expert human testing. As silver also notes:
The entire principle of understandable is critical for people with cognitive disabilities, yet success criteria intended to support the principle are not easy to test for or clear on how to measure.
Regardless of proficiency, there is a significant gap in how any two human auditors will identify a success or fail of criteria. … Ultimately, there is variance between: any two auditors; … Because there's so much room for human error, an individual may believe they've met a specific conformance model when, in reality, that's not the case. … There isn't a standardized approach to how the conformance model applies to success criteria at the organizational level and in specific test case scenarios.
based on best knowledge,for example -by monitoring and repairing non-conforming content within 2 business days; +
based on best knowledge,for example by monitoring and repairing non-conforming content within 2 business days; or
The provision of monitoring and required repair within a 2 business day window doesn't address the underlying challenge of pages with (3rd party) @@ -433,7 +406,7 @@
The
first Note under the definition of Accessibility Supported states that: The WCAG Working
-group and the W3C do not specify which or how much support by assistive
+group and the W3C do not specify which or how much support by assistive
technologies there must be for a particular use of a Web technology in
order for it to be classified as accessibility supported.
This is further expanded upon in the section
@@ -442,7 +415,7 @@
This topic raises the question of how many or which assistive technologies must support a Web technology in order for that Web technology to be considered@@ -1155,7 +1128,7 @@accessibility supported.The -WCAG Working group and the W3C do not specify which or how many +WCAG Working group and the W3C do not specify which or how many assistive technologies must support a Web technology in order for it to be classified as accessibility supported. This is a complex topic and one that varies both by environment and by language.
Reliably Human Testable,
not reliably testableIs Accessibility Conformance an Elusive Property? (Brajnik et al. 2012), found the average agreement was at the 70-75% mark, while the error rate was around 29%. +
Reliably Human Testable,
not reliably testableIs Accessibility Conformance an Elusive Property? (Brajnik et al. 2012), found the average agreement was at the 70-75% mark, while the error rate was around 29%.
accessibility supported ways of using technologies
accessibility supported ways of using technologies