Accessibility (computed role + computed label)

[zcorpan]

Description

Fix interop bugs in the browser’s accessibility tree for the parts that are now testable in wpt as a result of the interop-2023 accessibility testing investigation area, for the benefit of users who need assistive technology and for web developers wishing to design accessible user interfaces on the web.

Specification

https://w3c.github.io/aria/ https://w3c.github.io/accname/ https://w3c.github.io/html-aam/

Open Issues

No response

Tests

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in web-platform-tests/interop-accessibility#3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

Current Implementations

• Blink
• Gecko
• WebKit

Standards Positions

No response

Browser bug reports

No response

Developer discussions

No response

Polls & Surveys

in preliminary results from State of HTML 2023, ARIA was a common response to the question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

Also see accessibility_pain_points responses in the same survey: https://coda.io/d/State-of-HTML-Preliminary-Results_d7UVQ_fZW0V/Other-Core-Pain-Points_suDnF#Other-Core-Pain-Points_tuN-h/r4723

Inconsistent handling of elements/components between screen readers & browsers

As a FOSS developer, I can only test for browser/OS/AT combinations I personally own and that doesn't give me enough confidence

Most of the problems are UA problems - the UAs need to be quicker to fix bugs, so that developers who implement the specs can be confident they aren't doing the wrong thing

Browser support

Segmented support between browsers and OSes, have hit many cases I can support in 2 browsers but not a third. Safari and Firefox are the usual culprit.

Inconsistency between AT and browsers on how information is exposed.

Differences between browsers and tools (e.g. NVDA vs VoiceOver)

Difference between browsers, OS and screen readers

browser/AT interop

Different browsers expose different information to different assistive technology tools in different ways. Testing accessibility is a huge burden due to the discrepancies between the AT

Existing Usage

No response

Workarounds

No response

Accessibility Impact

Positive; this is explicitly designed to improve interop of some of the most important browser accessibility features.

Privacy Impact

No response

Other

No response

[foolip]

For survey data and web developer demand, in preliminary results from State of HTML 2023, ARIA was a common response to the freeform question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

[dtsengchromium]

I think there's issues with the tests themselves and the way they fetch data from Chrome. One issue I've written up follows.

WPT calls into webdriverio
https://github.com/webdriverio/webdriverio
in
webdriverio/packages/devtools/src/commands/getElementComputedRole.ts
accessibility.snapshot

which is Puppeteer
https://github.com/puppeteer/puppeteer
puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts
which sends the CDP command
Accessibility.getFullAXTree
with the interesting option defaulted to true.

In Chrome,
getFullAXTree
InspectorAccessibilityAgent::getFullAXTree
gets called and returns the tree back over CDP.

Back in Puppeteer,
puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts
Accessibility.snapshot

the tree is filtered for "interesting nodes". This amounts to calling into its own wrapper AXNode ts object isInteresting method.

That method has some definition of interesting which includes only some elements e.g. controls, leaves (with text), etc. That explains a lot of failures within the dashboard pertaining to roles.

I'm therefore not sure we can use the test results here until the underlying tooling is reliable/corrected. Specifically here, the simplest thing is for WPT to pass interesting to false
https://pptr.dev/api/puppeteer.accessibility.snapshot

[cookiecrook]

@dtsengchromium The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct? cc @aleventhal

[cookiecrook]

Though I could see ARIA Reflection (Gecko bug 1858211) being a problem for Gecko if there are objections to the test expectations or spec. If so, we could exclude that test from the Focus Area too.

[jcsteh]

[Edited by James: Oops, I accidentally corrected this one when trying to correct my own above. Sorry, this quotation is back to the original now. - jc]

@jcsteh The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct?

I don't see any comments from me on this particular issue, so I assume you're referring to web-platform-tests/interop-accessibility#51 (comment)? So no, those comments don't block this because it's out of scope.

You're correct that ARIA element reflection is a problem for us with regard to this focus area. This should be discussed somewhere else, but as I understand it, one of the primary use cases of that entire spec is to support references across shadow DOM, yet it doesn't actually achieve that goal because of the encapsulation problem.

There are a few other tests I think we should consider excluding from this as well for various reasons, but I'll provide more details on that once I've finished my triage.

[jcsteh]

There seem to be some tests where we don't have a clear path forward and no engine is passing. For example, html-aam/fragile/area-role.html and el-thead/el-tfoot/el-tbody in html-aam/table-roles.html. We'll either need to exclude these or collectively determine a path forward in order to get them passing.

[dtsengchromium]

@cookiecrook

My comment above was referring to the way in which trees were fetched for the tests and went into detail on the manner WPT does it. Happy to have that corrected if I missed something. The takeaway is that many of the results don't mean anything at the moment.

Separate questions are whether webdriver is the right component to base these tests off of and if so, whether its worth making contributions there. For Chrome, webdriver leverages puppeteer. Who wants to make contributions to those projects to get things working? IIRC, those projects primarily concern themselves with test automation.

Also, note that puppeteer is fetching Chrome's internal Blink side a11y tree. This is probably not the right tree to test against. For example, the role literals are internal programmatic identifiers and not stable. The name computation is incomplete since we do different things for each platform a11y api (grant it, in the minority of cases).

The platform trees are the only stable representation which we really service (e.g. NSAccessibility, IAccessible2, etc). These questions and others worth a discussion at the next meeting for a11y interop.

[boazsender]

I agree with @cookiecrook that web-platform-tests/interop-accessibility#51 is not part of this proposal as it's written. This proposal is about closing interop gaps from WPTs that don't use the axe tree.

@dtsengchromium what you're saying makes sense, we reached the same conclusion at Bocoup when we tried to use the available accessibility tree dump through cdp with the intention of testing core-aam and realized we couldn't for the reason you've mentioned here. I think we should continue this conversation on web-platform-tests/interop-accessibility#51 and discuss the best path forward for exposing the platform trees for testing aam.

I also started an issue here about continuing the 2023 accessibility investigation area into 2024, which is a place where I think we could continue this work if we don't advance it further this year: #459

So the tree dumping conversation would be part of the 2024 investigation area carry over, whereas this issue is a new focus area for closing interop bugs from tests based on things like getcomputedrole. I know it's a lot of interop jargon, so apologies for the category nitpicking. Where these things fall ends up being important for the wpt interop prioritization process. That's the only reason I'm pushing on this detail.

[zcorpan]

I think there's some misunderstanding here; as noted in web-platform-tests/interop-accessibility#65 (comment) , @dtsengchromium did not comment about tests using tree dump (which don't yet exist?) but about tests using get computed role (which is implemented using getFullAXTree for Chrome).

[boazsender]

Ah I understand now, thanks for clarifying @zcorpan.

@dtsengchromium what do you think would the best path forward for changing the underlying tooling?

[cookiecrook]

Yes, thanks for clarifying… FWIW, Chromium is already passing ~92% of the tests proposed for next year, but I can see the HTML-AAM: el-code test is failing as @dtsengchromium describes.

[cookiecrook]

@jcsteh wrote:

I don't see any comments from me on this particular issue,

Yes, sorry. That was a GitHub autocomplete that I mistakenly confirmed. Apologies as I meant to reply to @dtsengchromium

[dtsengchromium]

Fixing the tooling would involve upstream working in possibly each of WebDriver, Puppeteer. I don't know if this work could be rolled up into any other efforts/FAs?

[jensimmons]

We should figure out how to define the scope of the tests chosen for Interop 2024, and then set the scope. Not keep letting it grow over the course of the year.

[jugglinmike]

@dtsengchromium Thanks for that analysis! I wanted to understand the extent of the dependency on webdriverio, and I figured that I'd be able to find it in WPT since most of the "priority 1" tests which @zcorpan references have been completed. I searched WPT for references to webdriverio, but nothing came up:

$ git log -n1 --format='%h %ci %s'
b2528beea2 2023-11-01 11:05:04 -0700 [css-tables] Make `col-change-span-bg-invalidation*.html` tests fuzzy (#42897)
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

I thought maybe we're talking about the Chromium project's integration with WPT, but I couldn't find any references in Chromium's source tree, either:

$ git log -n1 --format='%h %ci %s'
a87e318028 2023-11-02 00:55:19 +0000 Cleanup "inverse-fieldtrial" trybots.
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

Would you mind pointing me in the right direction?

[dtsengchromium]

@jugglinmike I'm not too familiar with the WPT repo and how it's set up and it's been a couple of weeks. What you're looking for in WPT is the point at which it calls get computed role (I don't recall the exact method name). I traced down a couple of layers in WPT from the test html file to a common test ts file which appeared to lead to web driver.

It does also appear like maybe WPT has support for multiple test harnesses (e.g. servo, selenium, marionette).

[jugglinmike]

After tracing the flow of control, I've found a couple aspects that differ from what @dtsengchromium reported:

1. The tests appear to be instructing WPT's test runner to send WebDriver commands to Chromedriver (that is, neither webdriverio nor Puppeteer appear to be involved)
2. It seems as though the CDP command named Accessibility.getPartialAXTree is in use (not Accessibility.getFullAXTree)

These discrepencies are important because they impact how we'll fix the problem (and plan for the fix). I'll share the complete trace in the hopes that someone can verify my research or more precisely dispute it. (It's even more likely than normal that I've made a mistake because this infrastructure has been built with extension in mind.)

• the tests call AriaUtils.verifyLabelsBySelector
• AriaUtils.verifyLabelsBySelector calls test_driver.get_computed_label
• testdriver.js delegates to test_driver_internal.get_computed_label
• in WPT's test environment¹, wptrunner injects testdriver-extra.js
• test_driver_internal.get_computed_label calls create_action
• create_action pushes to __wptrunner_message_queue
• testharnessreport.js invokes __wptrunner_testdriver_callback with the message
• testharness_webdriver_resume.js defines __wptrunner_testdriver_callback as the argument...
• ...to the "async script" command of WebDriver in WebDriverTestharnessExecutor.do_testharness
• passing the result to WebDriverCallbackHandler.__call__
• which delegates to GetComputedLabelAction
• GetComputedLabelAction calls WebDriverAccessibilityProtocolPart.get_computed_label
• WebDriverAccessibilityProtocolPart.get_computed_label invokes WebElement.get_computed_label
• WebElement.get_computed_label sends the computedlabel WebDriver command
• ChromeDriver calls ExecuteGetComputedLabel
• ExecuteGetComputedLabel calls GetAXNodeByElementId
• GetAXNodeByElementId sends the CDP command Accessibility.getPartialAXTree

Footnotes

1. The Chromium project inserts a custom definition of test_driver_internal, but I haven't analyzed that path because we're considering the test results published on wpt.fyi. ↩

[cookiecrook]

Addressing the above subthread in this tracking issue.

• Chrome WPT failures may be due to incorrect implementation in ChromeDriver ExecuteGetComputedLabel interop-accessibility#65

[cookiecrook]

The fragile tests should be excepted from the Interop Focus Area proposal if web-platform-tests/wpt#39617 is not resolved.

• https://wpt.fyi/results/html-aam/fragile

[gsnedders]

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in web-platform-tests/interop-accessibility#3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

This leaves, what? https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative maybe?

[cookiecrook]

Perhaps also not path:/html-aam/fragile (I can rename that dir tentative if that's better)... I think it's unclear what the role should return for never-rendered elements like image map > area and the blocking issue is actually on the WPT test harness, not on an implementation.

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative%20and%20not%20path%3A%2Fhtml-aam%2Ffragile%20

[foolip]

I've taken a look at the test failures in Chrome to understand #526 (comment) and #526 (comment).

The 28 failing tests are:

/accessibility/crashtests/svg-mouse-listener.html
/accname/name/comp_embedded_control.html
/accname/name/comp_hidden_not_referenced.html
/accname/name/comp_label.html
/accname/name/comp_name_from_content.html
/accname/name/comp_text_node.html
/accname/name/comp_tooltip.html
/html-aam/fragile/optgroup-role.html
/html-aam/roles-contextual.html
/html-aam/roles-dynamic-switch.tentative.window.html
/html-aam/roles-generic.html
/html-aam/roles.html
/html-aam/roles.tentative.html
/html-aam/table-roles.html
/html/dom/aria-element-reflection.html
/inert/inert-inlines-around-selection-range-in-contenteditable.html
/svg-aam/role/roles-generic.html
/wai-aria/idlharness.window.html
/wai-aria/role/contextual-roles.html
/wai-aria/role/fallback-roles.html
/wai-aria/role/grid-roles.html
/wai-aria/role/list-roles.html
/wai-aria/role/listbox-roles.html
/wai-aria/role/menu-roles.html
/wai-aria/role/synonym-roles.html
/wai-aria/role/tab-roles.html
/wai-aria/role/table-roles.html
/wai-aria/role/tree-roles.html

Virtually all of these use aria-utils.js, which internally depends on test_driver.get_computed_role() and test_driver.get_computed_label(). The wpt.fyi Chrome results should end up using the implementation of this in testdriver-extra.js and executorwebdriver.py, which bottoms out in calling https://w3c.github.io/webdriver/#get-computed-role or https://w3c.github.io/webdriver/#get-computed-label.

@dtsengchromium, webdriverio or puppeteer are not involved here, and it's also not related to the work to dump the full accessibility tree. I wonder if there's been a crossing of the wires between this proposal and the investigation effort? I'm not very in touch with the details, but I hope I got the above right. It would be very useful to analyze the Chrome failures and confirm if they're real implementation bugs, or if there are indeed infrastructure issues that need resolving.

[cookiecrook]

@foolip @dtsengchromium

there is another issue tracking the Chrome-specific stuff here: web-platform-tests/interop-accessibility#65

[cookiecrook]

Accessibility WPT meeting: Group opinion is that the Chromium implementation issue is not a blocker for this Focus Area. The bug in ChromeDriver (presuming that's where the bug is) can be fixed as part of the effort to improve the Focus Area score.

[cookiecrook]

The SVG-specific tests were Priority 2 for the investigation and therefore excluded from this Focus Area tests, but of the 26 SVG tests finished for the investigation…

• WebKit passes 25 of 26.
• Chromium passes 25 of 26.
• Gecko passes 17 of 26 (failures mainly in the host lang label tests).

So we could leave it up to Mozilla to determine whether to include those 26.

[jcsteh]

I think it's reasonable to include these. I do have an open question about one of them (also failed by Chromium) in w3c/svg-aam#32 (comment).

[zcorpan]

@foolip is including the SVG tests ok for Chromium? (see above)

[jensimmons]

Tests of display:contents for consideration to include: web-platform-tests/wpt#43740

[dtsengchromium]

@foolip

Thanks for the analysis! My bad on the web driver, puppeteer paths; wasn't clear where things went in WPT.

It looks like the pruning may occur in several places. Will update here with more next week.

[cookiecrook]

@dtsengchromium No need to post that update here because it's not relevant to the Accessibility Interop Focus Area this issue is tracking.

There's another issue tracking the ChromeDriver bug in ExecuteGetComputedLabel. Please update that one:

• Chrome WPT failures may be due to incorrect implementation in ChromeDriver ExecuteGetComputedLabel interop-accessibility#65

[cookiecrook]

Minor adjustment to the issue title because not all the label stuff (img alt or html:label for example) that we are testing is defined in the AccName spec.

[cookiecrook]

As an aside, all the display: contents; accessibility tests from #568 are also only able to confirm testing via computedrole and computedlabel, so those tests could be rolled under this one if desired, as opposed to a separate Focus Area, but it seems fine either way.

[dandclark]

This proposal has been accepted as part of Interop 2024! On behalf of the entire Interop team, thank you for proposing it. You’ll be able to track progress on this topic throughout the year via the Interop 2024 dashboard.

[cookiecrook]

FYI some subtests in the tracked Focus Area files will be removed due to this bug:

• Remove orphaned children accessible tests, or clarify ARIA spec interop-accessibility#109

[zcorpan]

@cookiecrook or @nmlapre can you file a test change proposal?

[nmlapre]

can you file a test change proposal?

Sure thing - filed here: #645

[cookiecrook]

• Remove some accessibility tests due to new ARIA issue re: contents exposed vs nameFrom: contents #654