RFC: Accessibility APIs

[necolas]

This is a proposal to expand React Native accessibility APIs, and to align those APIs with equivalent web standards, e.g., WAI-ARIA. The aim is to provide developers with a single, familiar accessibility model rather than requiring deep knowledge of the models of each target platform.

[AgneLukoseviciute]

Have you thought about adding landmarks to this proposal? We're currently adding in support for this in RNW as it's quite important in desktop, might be something that could benefit mobile users as well.

[necolas]

Landmark roles are already accounted for in ARIA as values of the role attribute (equivalent to accessibilityRole prop). The additional ARIA roles (inc landmarks) to add to the RN API are included in this proposal and already supported by RN for Web. What where the reasons for deciding to introduce a separate prop for this in RN for Windows? The general idea here is to try to follow ARIA as closely as possible to avoid introducing proprietary props.

[FalseLobster]

Generally speaking I'm very much in favor of this idea! ARIA seems naturally suited to react-native for a number of reasons:

• It's not browser specific, if you read the WAI-ARIA spec they're very careful to use words like "user agent" to keep it agnostic of its primary use case today (accessibility in web browsers).
• The CORE-AAM does a great job of describing how ARIA attributes should map to native accessibility APIs, since it already is an x-plat abstraction for native accessibility APIs
• It's declarative, which suits react-native well
• The maintainers of the core-aam include representatives from Apple and Microsoft, who have a vested interest in ensuring ARIA can provide a good experience on their platforms. I think Google may be involved in the draft for the next version.

I think snapping to aria does raise a few questions, though:

• Does this proposal suggest react-native become a rigorous ARIA user agent, or just use APIs that similarly named? For example, the way accessible name and descriptions are computed is fairly complex and may involve some non-trivial native code on the various platforms code (out of tree and otherwise).
• Do you plan to deprecate accessibilityActions as part of the proposal? I didn't see it in the RFC. A number of native accessibility APIs are imperative in nature and accessibilityActions is a fairly nice way to add that functionality. It's not really possible to write a fully compliant Slider control on Windows using just ARIA because in ARIA all of the interaction is through keyboarding and simulated DOM clicks.

[necolas]

Does this proposal suggest react-native become a rigorous ARIA user agent, or just use APIs that similarly named?

Being pragmatic it's the latter for now. Although I think having specs for implementations is part of the appeal (as it is for Yoga to implement flexbox), and at worst the specs can be used to understand how the RN implementation differs from others.

Do you plan to deprecate accessibilityActions as part of the proposal?

That is the open question at the end of the proposal. In the short term these props are needed. If other changes to RN are made in the future we might be able to separate behavior from views, including platform-specific functionality like accessibility actions.

const ref = useAccessibilityActions({
  onAccessibilityAction,
  accessibilityActions
});

return <View ref={ref} />;

Problems like this lead me to ask: 1) Can accessibilityActions be inferred from ARIA/Web patterns? or 2) Would the Web benefit from APIs equivalent to the imperative native APIs accessed via accessibilityActions?

[AgneLukoseviciute]

Landmark roles are already accounted for in ARIA as values of the role attribute (equivalent to accessibilityRole prop). The additional ARIA roles (inc landmarks) to add to the RN API are included in this proposal and already supported by RN for Web. What where the reasons for deciding to introduce a separate prop for this in RN for Windows? The general idea here is to try to follow ARIA as closely as possible to avoid introducing proprietary props.

I hadn't realized landmarks in ARIA are implemented as values for role, thanks for pointing it out. That's the main reason I went with an implementation through a new prop, but we'll be able to add these values and move the logic over to accessibilityRole fairly easily.

[ksiler]

I'm also in favor of this proposal 🙂 my questions are:

• Have you considered adding properties accessibilityMultiSelectable and accessibilityRequired that would be equivalent to aria-multiselectable and aria-required? We currently support these concepts in react-native-win32 as part of the accesibilityState property and it's helpful for our native APIs when dealing with a group or list of elements that can be selected.

• What's the reasoning for removing accessibiltyHint in favor of just using accessibiltyLabel? I thought these properties had distinct purposes. How should developers convey extra information about a control if accessibilityHint is removed?

• To clarify, is the proposed accessibilityLevel property meant to refer to both text heading levels and levels of elements in a hierarchical structure (which can be inferred by the role)? I'm asking because we already have an accessibilityLevel property for react-native-win32 but it's just used for the latter scenario in conjunction with accessiblityPosInSet and accessibiltySetSize. We could easily repurposed it for identifying text heading levels too but I wanted to check if both scenarios fit the intended use.

[necolas]

Have you considered adding properties accessibilityMultiSelectable and accessibilityRequired that would be equivalent to aria-multiselectable and aria-required? We currently support these concepts in react-native-win32 as part of the accessibilityState property and it's helpful for our native APIs when dealing with a group or list of elements that can be selected.

Good to know you already need these props too. All the ARIA properties are supported in RN for Web and if we were to adopt the ARIA APIs they could be added to RN. I can include them in the proposal. There are even more ARIA attributes for defining tabular data, etc...do you also have a need for those yet?

What's the reasoning for removing accessibilityHint in favor of just using accessibilityLabel? I thought these properties had distinct purposes. How should developers convey extra information about a control if accessibilityHint is removed?

The accessibilityHint prop is an iOS-only concept. There is an attempt to implement it in RN for Android, but it's hacky and even introduces accessibility issues related to localization. There doesn't appear to be a clear need for this prop as the label should confer the necessary information and exists across all major platforms. Furthermore, the "additional information" use case is served by aria-describedby.

(This proposal is an attempt to provide guiding principles - ARIA - for defining new a11y props so we can limit further proliferation of adhoc prop additions.)

...is the proposed accessibilityLevel property meant to refer to both text heading levels and levels of elements in a hierarchical structure

Yes I think we should support both since that is the attribute's purpose in ARIA too. But it depends on the constraints of various native platforms and requires input from Android/iOS experts.

[blavalla]

While I like the general idea of having a more unified API between native and web, there are some pretty big differences that would need to be taken into account that would ultimately make this new API not quite close enough to web to be a direct mapping, but not quite close enough to native to also be a direct mapping, basically ending up as a strange hybrid of the two (which doesn't even take into account the large differences between how iOS and Android handle some of the basics like focus).

To give some context, web accessibility has a very declarative API, where you declare various attributes on DOM elements, and then allow the assistive tech (AT) to parse those out and determine its behavior. On native, while many properties are declarative (traits, labels, etc.), there is also the concept of directly interfacing with the AT itself to control it. Things such as sending accessibility events on Android, creating custom rotors on iOS, and making manual announcements (both platforms), all rely on the app directly telling the AT to do something, and it responding to this.

This difference is likely due to the significantly simpler interaction models that mobile ATs have, where a user may only have a few gestures they can use to control them, so much of the advanced control relies on the apps themselves to add specific actions and features for the user.

As for the details of this proposal, I have a few items that I think need to be thought through in more depth. I'll outline them by phase below:

Phase 1

• Consolidating accessibilityElementsHidden and importantForAccessibility works for common cases, but not more advanced ones. On Android for example you may want to set importantForAccessibility to "no_hide_descendants", or "no", one of which blocks this element and all descendants from this node onwards from being accessible, and the other one only blocks this specific element. If this became a boolean this functionality would be lost. This also disallows the setting of this value to "yes", although admittedly that has far fewer use cases (though not zero).
• Replacing the "adjustable" role with "slider" doesn't make much sense from an iOS perspective. Many non-slider elements are considered "adjustable", and this role maps directly to the UIAccessibilityTraitAdjustable trait, which is used for a number of interfaces where a user adjusts a value. In general, there is a real problem right now with the combining of "roles" (from web and Android) and "traits" (from iOS). Roles only describe what an element is, while traits are used to both describe what an element is and also how an element works. Trying to use one prop for both of these concepts means that there will always be strange edge cases like this.
• accessibilityHint is a very important feature on mobile, and can not be replicated by simply concatenating content into the label. Hints are purposely delayed after the main focus announcement, as they are meant to describe the result of taking an action on the element. A user should not need to hear them every time they land on an element, as they would become annoying quick quickly in UIs that a user understands. They are meant to teach the user how the UI works, and be easily ignored (hence the delay) if they are no longer needed. Android does have a concept of hint, but on Android hints are directly connected to accessibility actions, so a hint can't be a standalone string like on iOS, but must be associated with an action such as "click". We surface these in RN right now via the accessibilityActions API due to this required association with an action.

Phase 2

• I think we'll need to flesh out what these props actually do on mobile, as many of them either have similar equivalents already (posInSet compared to accessibilityValue min/max/now) or have no mobile counterpart (describedBy, activeDescendant, etc.).
• The biggest issue with the Phase 2 items is the concept of redefining "focusable" semantics. Android can map very closely to web here, but iOS simply does not allow a focusable element to have any focusable descendants. There is no way around this without basically re-creating an entirely custom hierarchy of accessibilityElements (that is still in a flat list, but ordered in a way that makes it seem like you are navigating to descendants) that is not connected to the actual view hierarchy.

Phase 3

• My only feedback here is that you've removed accessibilityIgnoresInvertColors without any sort of way to get this feature back via some other API. So this would just be a feature loss, that while not the most commonly used feature, has no other viable approach to fall back on if you do need it. The other changes all have had alternate APIs suggested, so their removal is okay assuming that those replacement APIs are well fleshed out.

So in conclusion, I really like the idea here, but I think it likely needs to have someone with expertise on each platform (web, Android, iOS, Windows) to really be involved to make sure we don't miss anything or leave any weird gaps. It's certainly possible to make a more unified API, but that API may not map as closely to one platform (web) as initially thought.

Let me know if you want details on any of the above. I would consider myself an expert in Android accessibility that can speak to any part of that system, and knowledgeable enough about iOS accessibility to make high level suggestions, but not enough of an expert on iOS to give in depth explanations of the technical details required to pull something off.

[necolas]

Thanks Brett. Are there any other native a11y APIs that are currently unavailable in RN?

On native, while many properties are declarative (traits, labels, etc.), there is also the concept of directly interfacing with the AT itself to control it. Things such as sending accessibility events on Android, creating custom rotors on iOS, and making manual announcements (both platforms), all rely on the app directly telling the AT to do something, and it responding to this.

If developers need to understand and manage 3+ different a11y APIs for general a11y tasks while using React Native, that would be a serious problem. What we have to decide is a) whether the differences are significant enough that the imperative controls cannot be abstracted behind the ARIA declarative API, and b) whether the use cases are common enough that they must belong in the xplat API (but only be supported by 1 or 2 platforms). There will always be a need for platform-specific APIs in specific scenarios, but I think we should avoid putting those APIs into the xplat layer. We have other mechanisms, such as custom components and hooks, by which platform-specific behaviors could be introduced.

On Android for example you may want to set importantForAccessibility to "no_hide_descendants", or "no", one of which blocks this element and all descendants from this node onwards from being accessible, and the other one only blocks this specific element.

Is this not mostly equivalent to aria-hidden (hides tree) and role="none" (removes semantics from individual element)? See MDN.

Replacing the "adjustable" role with "slider" doesn't make much sense from an iOS perspective.

The slider role isn't intended only for visual sliders, but for UI where a value is selected from within a range. There may well be differences between traits and roles, but RN has already combined the 2 concepts.

accessibilityHint is a very important feature on mobile, and can not be replicated by simply concatenating content into the label...We surface these in RN [for Android] right now via the accessibilityActions API due to this required association with an action.

Concatenating the content into the label is exactly what we do for the Android implementation of accessibilityHint, and since it's not done in user space this is a big problem for localized content. aria-describedby appears to be the rough equivalent of hints, in terms of the purpose and precedence it takes. If we were to keep accessibilityHint it would be as an iOS-only prop. Longer-term I don't think the accessibilityActions API belongs on the component props, but as a behavior hook. This proposal doesn't suggest removing that API at this stage though.

I think we'll need to flesh out what these props actually do on mobile

Agreed. If we were to do this it looks like we'd be committing to building certain abstractions over native APIs as browsers do. I have no idea what that involves on the native engineering side.

iOS simply does not allow a focusable element to have any focusable descendants

Technically neither does web. This is something we should enforce at the framework level in any case.

you've removed accessibilityIgnoresInvertColors without any sort of way to get this feature back via some other API

This is another iOS-only prop that has no relation to ARIA concepts. I'm not sure what the best replacement would be. I think we could either make iOS behave like other platforms at the framework level (maybe that means media is never inverted?), or perhaps model the API to control this on web equivalents (e.g., the forced-color-adjust style).

[necolas]

RE: Accessibility Actions

This Apple introduction to custom rotors is interesting: https://developer.apple.com/videos/play/wwdc2020/10116/

The first example of the categorized map icons would typically be implemented on web as buttons in either a list of lists, or lists below headings. VoiceOver users can then use the rotor to navigate by category headings. Other examples could perhaps be implemented by React Native, mapping web markup patterns to inferred custom rotors on native platforms. While the rest don't have clear equivalents.

There's been some discussion in the W3C ARIA GitHub about how to model these accessibility actions, suggesting they have similarities to custom context menus for accessibility services: w3c/aria#762

I suspect that the built-in accessibility actions we have in React Native could be surfaced to RN devs as events, e.g., we add onCopy, onPaste props, etc. These actions are needed outside of the accessibility contexts too - https://reactnative.dev/docs/accessibility#accessibility-actions. As the docs say, "Typically, this should perform the same action as when the user...[is] not using an assistive technology".

[ksiler]

@necolas We're currently looking at adding support for defining text heading levels in react-native-win32 by reusing the existing accessibilityLevel prop we have defined which is also referenced in the proposal here. Since I last asked about this over a year ago, I wanted to reconnect and verify we could move forward with this. Here is our previous conversation:

...is the proposed accessibilityLevel property meant to refer to both text heading levels and levels of elements in a hierarchical structure

Yes I think we should support both since that is the attribute's purpose in ARIA too. But it depends on the constraints of various native platforms and requires input from Android/iOS experts.

Are there any android/iOS contacts I should reach out to to discuss this? Or is the proposal outlined for accessibilityLevel approved?

[necolas]

yes I think it's ok. that prop is only implemented by the windows and web forks, and for the same purpose

[blavalla]

Just as something worth noting, I don't think there is any Android equivalent to heading levels, so this property likely will be a no-op on that platform. The only place that Android has any sort of level information like this is with AccessibilityCollectionInfo, which has a boolean isHierarchical saying whether this list should be treated as hierarchical content and whether nested lists should be considered as different levels.

There's no control over the actual level other than the depth of the list though, so even if this was set on elements within hierarchical list structures it wouldn't really be applicable.

[necolas]

Closing this in favor of #496