Accessibility (A11y) #167
Comments
|
From the top of my head, here are some tasks to improving accessibility:
The accessibility data needs structure ("there is a window here with a scroll area") and semantics ("this is a heading, this is a checkbox"). What are the good, open standards for such accessibility data? Is there an accessibility API for the web that doesn't require Ideally I'd like egui to just output the GUI structure as e.g. JSON and pipe that through some accessibility API. I'd appreciate some help with this as I don't have any experience here! |
emilk
added
design
|
@emilk There's some potentially helpful content in this Bevy issue:
|
|
Thanks for those links @follower ! |
|
I'd hoped I might've been able to find some Free/Open Source Software developer specific accessibility tools/guidelines[0] but didn't manage to find much from a brief search.
There's a reasonable amount of coverage of WWW/HTML/DOM aspects but less so for Canvas/desktop applications. (There's some older somewhat "enterprisey" content but it's a little difficult to separate the wheat from the seo content marketing...) A couple of links that might still have some applicability: Additional thoughtsThere definitely seems to be opportunity for egui to have accessibility as a strong part of its story which has the potential to be compelling for both legally mandated & philosophical reasons--and seems consistent with the values of the Rust ecosystem itself. Of course, if it were easy everyone would be doing it, I guess... Based on the thread I linked above perhaps the recent Bevy+egui integration might be a good place to explore accessibility-related development further. As I mentioned in one of the comments in that thread (while quoting the
Hope some of the linked resources might be useful. More linksUpdate: I may have subsequently gone a little overboard :D but figured I might as well put these here too:
Update (August 2021):
|
|
I'd like to point out that I think this is a very important issue, but I fear it is also a pretty large task. So far egui is only a hobby project, and I have just a few hours per week to spend on it, so any help here would be greatly appreciated! |
|
[Started writing this almost a week ago so just wrapped it up quickly to finally get it posted. :) ] Appreciate you providing that context, @emilk. At a meta level, based on what I've learned (hopefully correctly :) ) from other communities with regard to accessibility & inclusiveness, I'm conscious of these aspects:
(And, while in my mind a site/repo acting as "one stop shop" on "how to make my project more accessible & inclusive" seems beneficial, for now I'm limiting myself to just documenting in this issue what I learn. :D ) Will follow-up with further specifics. |
Existing information, experience & resourcesIn light of (1) above I decided to revisit the work done in https://github.com/lightsoutgames/godot-accessibility to see what I could learn; and, (re-)discovered that the included Godot-specific plugin https://github.com/lightsoutgames/godot-tts (see) is actually built on top of a Rust crate https://crates.io/crates/tts (see & see) (all developed by @ndarilek).
|
|
BTW @nvzqz do you have a specific evaluation criteria/check-list in mind that could serve as a framework/target during design/development? |
|
Also, a couple of relevant recent items discovered re: GTK4 & Accessibility:
|
|
Hey, this is exciting work. Thanks for pinging me, and for putting
together this PoC. I'd earmarked this for when I needed to investigate
UI for my game, but covering the same accessibility ground is a lot less
enjoyable than making games. :) So thanks for the start.
Re: TTS chattiness. I haven't looked at your code yet, because as you've
noted we're having a once-in-a-generation winter storm here and I'm
stuck on a 12-year-old laptop with no modern graphics hardware. :) But
one quick mitigation is to use the second `interrupt` argument to
`speak` and set it to `true` at the start of each utterance. So whenever
focus moves to a new widget, speech is interrupted and TTS is less
chatty. Also do that for typing characters into text fields. Screen
readers have additional hooks, but those are hard to make generic. Maybe
you've done that already--opening any Rust codebase in VSCode grinds
this poor thing to a halt.
I'll look into this and help out more when I'm back on my main
workstation in an apartment with reliable power. :) Thanks again!
|
|
Finally getting some time to play with this.
I'm prototyping in Bevy because it's a bit clearer to me how I might
begin separating out the screen reader functionality from the UI itself.
I notice that some members of `interaction`, particularly `kb_focus_id`,
seem to have changed from `pub(crate)` to `pub` in 0.10. Is there some
way to access `kb_focus_id` from memory? I've never worked with
immediate-mode GUIs before. I suspect the way to monitor focus for
changes is to have a function/system check for changes in my game loop,
and act accordingly? If so, is there some way I can access the focus ID?
I thought about PRing a getter on `Memory`, but wanted to check whether
I'm missing something else first. Also wanted to make sure such a PR
would be welcome.
Also, is there some way to get a widget based on its ID? Once I'm
successfully tracking changes, I'm going to want to figure out which
widget is associated with them and act accordingly. The prototype does
this a bit roughly, but now I'm wondering what it'd take to make this
pattern more broadly applicable.
Thanks.
|
|
@ndarilek There is So one design would be that widgets emit an event to ui.memory().interested_in_kb_focus(id);
if ui.memory().gained_id_focus(id) {
ui.output().events.push(OutputEvent::WidgetGainedFocus(WidgetType::TextEdit, text.clone()));
}
Widgets are pieces of of code that is run each frame. |
|
#31 has been closed - you can now move keyboard focus to any clickable widget with tab/shift tab. Next up: I'm gonna add some outgoing events from egui every time a new widget is given focus. That should then be fairly easy to hook up to to TTS system. |
|
egui now outputs events when widgets gain focus: https://github.com/emilk/egui/blob/master/egui/src/data/output.rs#L56 This should be enough to start experimenting with a screen reader, and should provide a framework for building more features around. There's still a lot more to do! |
|
Background: I built one of the first open source Android screen readers,
so I have some sense for how accessibility APIs should work.
I recognize the challenges immediate-mode GUIs pose. I think, though,
that there does need to be some sort of central registry of widgets
independent from the GUI code, queryable by ID. I need a single
entrypoint that'd let me, say, capture every exposed event on every
created widget and perform some sort of accessibility handling, possibly
querying for other widgets via ID as well. Without that, we'll either
need lots of code duplication, or special handling of every widget at
time of creation. The harder we make accessibility, the less likely it
is to just be done.
I wonder if some sort of HashMap<ID, _> would be sufficient? Widgets are
automatically added at time of creation. Create a `Drop` impl that
cleans up the registry whenever the widget goes out of scope.
Seems doable, with the possible caveat that whatever gets stashed in the
registry needs to be a weak reference that knows to clean itself up
whenever it's the only reference to a given widget. The events would
then ship along a widget ID, because whatever presents them is going to
need to access very specific properties on each widget type. Not sure
how doable that is under the current system.
You can see an example of what I mean at
https://github.com/lightsoutgames/godot-accessibility/blob/master/Accessible.gd#L588.
Maybe there's a cleaner way to do that, but essentially I spawn a Godot
`Node` under every control, and make it do a whole bunch of
introspection on its parent to present it sensibly. After a few tree
scrapes to guess labels for things like text fields, it then performs a
giant conditional check and runs a whole bunch of subordinate functions
for widget-specific logic. And those functions each need deep knowledge
of whatever widget they're operating on.
Thanks for being willing to experiment with this! :)
|
|
@ndarilek thanks for helping out! The problem with storing references to widgets is that in immediate mode, widgets is not data, but code that is run once per frame. See for instance the As for events: I've just added so that You can checkout latest
I'm gonna try hooking this up to a simple TTS system in |
I almost posted my 2 cents here yesterday on exactly this topic. I believe that immediate mode GUIs can build an in-memory representation of the UI as a DAG, just as easily as a retained mode GUI can. This DAG can be queryable, individual elements can provide additional accessibility context, and user code can go the extra mile to provide application-specific context as needed.
egui/egui_demo_lib/src/apps/demo/tests.rs Lines 21 to 33 in 44cd304 This is necessary for several reasons, but the same trick can be used on each frame to create the structures necessary for interacting with accessibility APIs. This is a design I have been kicking around in my head for quite a long time. I like the flexibility and no-nonsense approach to immediate mode GUIs, but I am also aware of (some of) the needs of accessibility software in relation to GUIs. Something of a hybrid immediate/retained approach is the best-of-both-worlds; The API remains immediate, and some state is retained for ease of use and doubles as a source of truth for screen readers. |
|
If you checkout This is still very early, and more events needs to be hooked up (e.g. you probably want to hear a voice when editing a label and not just when first focusing it). There is one more thing to consider: how should an egui app know whether or not to read things out loud? I can't find a javascript function for "does the user want a screen reader". The same problem exists on native, but I'm sure with some digging one can find platform-specific ways of doing so. |
|
@parasyte Having egui keep a shadow-DOM behind the scenes is a big change though, and requires a lot of work and planning. Before doing that I'd like to be absolutely sure we need it. In my proof-of-concept screen reader I get away with describing just the newly focused widget so there is no need to store and manage a bunch of state behind the scenes. |
|
While I'm not a user of assistive technology, I want to bring up a point: I think "implementing TTS" is the wrong way of looking at the problem. Someone using a screen reader already is using a screen reader, whether it be Microsoft Narrator, macOS VoiceOver, or whatever. Support for screen readers thus isn't implementing TTS readouts for your own window, but letting existing screen readers effectively read your window. I don't know what that entails, to be completely honest. I just want to make sure that we're not accidentally going the wrong direction (which I kinda got vibes from "how do we tell if the browser wants TTS"). |
|
Agreed, but that problem is a lot more complicated than a single, or even a small handful, of developers can manage, and is further complicated by immediate mode. AccessKit aims to solve it in a cross-platform way, and when that's available, an egui integration might be practical. But until that happens, implementing "real" accessibility is likely beyond us. |
|
AccessKit, which @ndarilek mentioned in the previous comment, is now far enough along that we can start working on integration into egui. I have a very basic integration of AccessKit into egui on this branch. Here's a quick summary of how AccessKit works, and how it fits into egui. AccessKit takes a push-based approach to accessibility. That is, for each frame where something in the UI has changed, the application creates a TreeUpdate, which can be either a complete tree snapshot or an incremental update, and pushes it to an AccessKit platform adapter. That platform adapter can then handle requests from assistive technologies (e.g. screen readers) without having to call back into the application, except when the user requests an action such as changing the keyboard focus or doing the equivalent of a mouse click. So in principle, this model is a good fit for an immediate-mode GUI. (In practice, the implementation could probably be made more efficient, e.g. by eliminating repeated heap allocations.) My integration creates a complete AccessKit tree for every egui frame, and AccessKit does comparisons to figure out what actually changed and fire the appropriate events. AccessKit itself is still far from complete, and so is the integration. Most notably, I still need to work on support for text edit controls, as well as reading the value of a slider, and lots of smaller stuff. Also, AccessKit is only implemented for Windows so far. Still, at this point, you can run the eframe hello_world example on Windows, start up any Windows screen reader (Narrator, NVDA, JAWS...), tab around and get feedback, or navigate with the screen reader's commands. AccessKit and egui support one screen-reader-initiated action so far: setting the keyboard focus. It won't be hard to implement more. I've modified egui-winit to use a proof-of-concept integration of AccessKit into winit which I've posted in my fork of that project. That direct integration into winit isn't likely to be accepted upstream, so I'll ultimately have to come up with another solution for that part. It's also worth discussing how this work should relate to the existing "widget info" support. My AccessKit integration into egui currently uses the widget info. But another option would be to have all of the widgets manipulate AccessKit nodes directly, implement a generic, egui-independent screen reader library that uses the AccessKit tree, and ultimately drop widget info from the output struct. We're going to need direct text-to-speech output for a while yet, until AccessKit is implemented on all of the other platforms. (And even then, self-voicing would be useful for devices with no built-in screen reader, like game consoles.) But perhaps egui itself shouldn't have two ways of doing accessibility. |
|
Wow @mwcampbell that's sounds great! |
|
A quick update: AccessKit is still Windows-only, and there are still serious limitations in the Windows implementation, most notably lack of support for text editing. But one major blocker has just been resolved: the newly published accesskit_winit crate makes it straightforward to use AccessKit with winit, without requiring changes to winit itself. I'm aware that my fork of egui with prototype AccessKit integration is way out of date. My next task is to update it and use the new winit adapter rather than my forked winit. |
|
@emilk On #1844, I mentioned the possibility of replacing egui's current |
|
The accesskit branch in my egui fork has a rough but basically working AccessKit integration. It's based on the egui master branch as of earlier today. The other key difference between this branch and the work I did last December (which is now in the accesskit-old branch) is that I'm no longer using a fork of winit. (In fact, all dependencies are published crates.) Currently, The big missing feature, still, is text editing. I'm starting on that in AccessKit next week. Aside from that, egui still needs to expose some more things that are already supported by AccessKit, such as the value and range of a slider. And, for now, AccessKit is still only natively implemented on Windows. That's changing later this year. In the meantime, a platform-independent embedded screen reader, which is what accessible egui-based applications currently have to use, can be written based on AccessKit, using the accesskit_consumer crate to process the tree updates, traverse the tree, and find out what changed from one frame to the next. |
|
@mwcampbell thank you so much for your work on AccessKit, and on working on the egui integration! I like your current approach of having The egui screen reader is mostly a proof-of-concept, and I don't believe it has many users right now, so breaking that end of things is less worrying to me. Still, if given a choice I would keep I took a look at your egui fork, and it looks great so far. But, I would prefer having |
|
I just rebased my accesskit branch on the head of the egui repo (as of yesterday). @emilk Do you want AccessKit integration to be an optional feature at all layers, including eframe, or just in the core egui crate? I'm also wondering if the accesskit_winit adapter should be integrated in egui-winit, as it is now in my branch, or only in eframe. The latter would reduce the total PR size, but would mean that anyone using egui-winit but not eframe would have to do more work to get AccessKit support. |
|
As a developer user of egui, eframe is framework enough that it makes sense IMHO to always have AccessKit enabled. At most, it could be a default feature; people should be pushed towards accessibility by default. For egui and egui-wgpu, though, it's a major use case to use egui for development debug UI on top of your own rendering, and in that case it's likely the case that a graphics focused application wouldn't want the default integration as it'd likely need a fully custom solution to be accessible via screen reader. As such, my vote goes to having an opt-in feature for AccessKit in egui/egui-wgpu with a separate entry point to turn on the integration, but that it should be as simple as using the ak enabled initialization. I don't know how practical that is, but it's what I'd personally like using. |
|
AccessKit is now an optional dependency in the core egui crate. I won't do anything more with egui_winit, egui_glium, egui_glow, and eframe until I get more input from @emilk. |
|
I'd also like input on what milestones I should reach before I submit my AccessKit integration as a PR. AccessKit still only has a Windows adapter, though adapters for other platforms are now being developed (by others), and the accesskit_winit crate uses a no-op implementation on non-Windows platforms. Meanwhile, the core AccessKit crate hasn't yet reached 1.0, and I'm not sure when it will. The biggest missing functionality at the moment is text editing support. I'm hoping the API for that will be close to its final form by sometime next week. On the one hand, if I wait until everything is done and perfect before I submit a PR, then I'll need to keep maintaining my own branch, and rebasing it on new versions of egui, for a while. On the other hand, I don't want to impose on the egui team the burden of keeping up with changes to AccessKit too soon. |
|
You can open a draft PR right away @mwcampbell - it will make it easier for me to review your work! |
|
OK. I'm currently in the middle of working on text editing, both in AccessKit and in my egui branch. Once I finish that and get sliders working, I'll open a draft PR. |
|
If anyone wants to play with my work-in-progress text editing support with a Windows screen reader, here's the egui branch. Note that the AccessKit side of this is still a work in progress and isn't in the published crates yet; here's the AccessKit branch. At this point, the major missing feature in text editing support is that the bounding rectangles of text ranges aren't yet exposed. This is why, if you use Narrator, the highlight cursor isn't where it should be when you're in a text edit widget. I suspect it's also why the JAWS cursor isn't working. I plan to implement this today. Also, Narrator isn't providing the expected feedback when deleting text; I'm guessing that's because AccessKit is returning an inappropriate error when Narrator tries to work with the old text range. I'll also look at this today. Once I resolve those two issues, I'll open a PR on the AccessKit repo. Once that's merged, I can merge the egui work back into my main AccessKit branch. |
|
I just released text editing support in AccessKit (still Windows only), and the matching support in egui is now on my main accesskit branch. I'm going to rebase that branch to the head of the upstream master branch, then I think I'm ready to open a draft PR. |
|
FWIW, @DataTriny is working on a Linux platform adapter for AccessKit, implementing AT-SPI in pure Rust. He thinks that might be usable by the end of the year. Once that feature is merged in AccessKit, and AccessKit support is merged in egui, I think we will no longer need speech-dispatcher on Linux. That dependency seems to be a recurring source of frustration for egui developers. |
|
If anyone wants to try out the work-in-progress AccessKit macOS adapter, check out this temporary egui branch. The major missing features that you're likely to encounter in simple example apps are:
I plan to address the first two early next week. Text editing will likely take several days. I hope to have the macOS adapter on par with the Windows one by early December. I'm posting a status update on this here because I know that macOS is popular among developers, and I figure that when AccessKit's macOS adapter is reasonably complete, interest in AccessKit in general will increase. |
|
Quick update for anyone watching this issue but not #2294 (AccessKit integration PR): I've marked that PR as ready for review. That means I've frozen the initial implementation, except to address review feedback. On the macOS side, that adapter is now published and is used by the published AccessKit winit adapter. So my egui AccessKit integration now supports Windows and macOS using only published AccessKit crates. I resolved the hit-testing and slider/stepper issues mentioned in the previous comment. Now the big feature I need to work on for macOS is text editing support. |
|
I just want to say: Thank you @mwcampbell for your work here! Your efforts are greatly appreciated. 😄 |
|
Thanks to @emilk for merging AccessKit support. To be clear, this doesn't mean the end of all work on accessibility in egui. AccessKit itself is still incomplete, and not all widgets are fully accessible yet. But I think this is an appropriate time to close this original issue and open new ones as they come up. @emilk Feel free to reopen if you disagree. |
|
I agree with closing this, and I also agree with @CAD97 - thanks for working on this @mwcampbell ❤️ |
Is your feature request related to a problem? Please describe.
Not all GUI app users are sighted or can see well. This project does not indicate any effort to make apps accessible to the visually impaired.
Describe the solution you'd like
For this project to make a conscious and focused effort to support apps that can be used by the visually impaired.
Describe alternatives you've considered
None.
Additional context
I (and many others) will not use a GUI library in production unless it is designed with accessibility in mind.
The text was updated successfully, but these errors were encountered: