Accessibility: Screen readers doesn't announce any information on the connected nodes or direction of connection

[amitkumariiit]

Describe the bug
Opening this new one as old one was closed. We checked on version 8.12.0 and its not fixed.

Accessibility: Screen readers doesn't announce any information on the connected nodes or direction of connection.

To Reproduce
Steps to reproduce the behavior:

1. Go to Flowchart
2. Launch screen reader (e.g narrator or NVDA) and navigate to the right window using caps+arrow key.
   

Expected behavior
Screen reader should announce more detailed information of the image ,there should be an alternative text provided describing the rendered svg.

Actual:
On navigating to the image using caps+arrow key , the narrator announces start and stop only , there is no information of it as a connected nodes or the direction of connection.

User Impact:​
The screen reader users will not receive clear context about the image present on the screen.

Screenshots
If applicable, add screenshots to help explain your problem.

Code Sample
If applicable, add the code sample or a link to the live editor.

Desktop (please complete the following information):
OS: [Windows]
Browser [Edg/Chrome]
Version [NVDA/Narrator]

Additional context
Add any other context about the problem here.

[aardrian]

I want to add some more context to this issue.

Using the following flow chart:

  flowchart TD;
      A[Using a flow<br>chart in GitHub] --> B{Must it be<br>accessible?};
      B -- No --> C[Wrong,<br>try again];
      B -- Yes --> D{Into what format<br>does it render?};
      D -- JPG/PNG/GIF --> E{Does it have<br>an `alt`?};
      E -- No --> V[Wrong,<br>try again];
      E -- Yes, and it<br>is lengthy --> V[Wrong,<br>try again];
      E -- Yes, a<br>brief one --> F[Manually add<br>a plain text<br>description<br>of the chart.];
      D -- SVG --> G{What `role`<br>does it have?};
      G -- `none` --> N[Wrong,<br>try again];
      G -- `table` --> N[Wrong,<br>try again];
      G -- `grid` --> N[Wrong,<br>try again];
      G -- `presentation` --> Y[Manually add<br>a plain text<br>description<br>of the chart.];
      G -- `image` --> H{Does it have<br>an `aria-label`?};
      H -- No --> P[Wrong,<br>try again];
      H -- Yes, and<br>it is lengthy --> P[Wrong,<br>try again];
      H -- Yes, a<br>brief one --> Z[Manually add<br>a plain text<br>description<br>of the chart.];
      G -- `tree` --> I{Does it contain<br>any `treeitem`<br>roles for the<br>text nodes?};
      I -- No --> R[Wrong,<br>try again];
      I -- Yes --> J{Are they contained<br>in / owned by `group`<br>roles beyond the<br>first level?};
      J -- No --> S[Wrong,<br>try again];
      J -- Yes --> K{Have you<br>tested in<br>any screen<br>reader?};
      K -- No --> T[Wrong,<br>try again];
      K -- Yes --> L{Did it accurately<br>convey relationships<br>and structure from<br>the chart?};
      L -- No --> U[Wrong,<br>try again];
      L -- Yes --> M[You should write a<br>CSUN talk about it];

Loading

Screen readers announce it as nothing but unordered run-on text. There are no relationships, no structure, nothing to give context to the chart. It lacks necessary roles and properties as well.

Here is the output directly from the speech viewer of JAWS 2021 with Chrome (the line breaks are from the <br> elements in the text):

Unlabeled 0NoYesJPG/PNG/GIFNoYes, and it
is lengthyYes, a
brief oneSVGnone``table``grid``presentation``imageNoYes, and
it is lengthyYes, a
brief onetreeNoYesNoYesNoYesNoYesUsing a flow
chart in GitHubMust it be
accessible?Wrong,
try againInto what format
does it render?Does it have
an alt?Wrong,
try againManually add
a plain text
description
of the chart.What role
does it have?Wrong,
try againManually add
a plain text
description
of the chart.Does it have
an aria-label?Wrong,
try againManually add
a plain text
description
of the chart.Does it contain
any treeitem
roles for the
text nodes?Wrong,
try againAre they contained
in / owned by group
roles beyond the
first level?Wrong,
try againHave you
tested in
any screen
reader?Wrong,
try againDid it accurately
convey relationships
and structure from
the chart?Wrong,
try againYou should write a
CSUN talk about it

Here is output from the speech viewer of NVDA 2021.3 with Firefox (same about the line breaks):

No Yes JPG/PNG/GIF No Yes, and it
is lengthy Yes, a
brief one SVG none table grid presentation image No Yes, and
it is lengthy Yes, a
brief one tree No Yes No Yes No Yes No Yes Using a flow
chart in GitHub Must it be
accessible? Wrong,
try again Into what format
does it render? Does it have
an alt? Wrong,
try again Manually add
a plain text
description
of the chart. What role
does it have? Wrong,
try again Manually add
a plain text
description
of the chart. Does it have
an aria-label? Wrong,
try again Manually add
a plain text
description
of the chart. Does it contain
any treeitem
roles for the
text nodes? Wrong,
try again Are they contained
in / owned by group
roles beyond the
first level? Wrong,
try again Have you
tested in
any screen
reader? Wrong,
try again Did it accurately
convey relationships
and structure from
the chart? Wrong,
try again You should write a
CSUN talk about it

The announcement is a bit different than what is shown here (every ` announces as "grav", for example). As you read the preceding text, I am sure you can understand how that conveys nothing of the meaning of the chart.

Some of the reasons this is inaccessible:

• the overall SVG exposes no structure;
• the SVG nodes have no semantic or structural meaning;
• the connectors have no source order position that corresponds to the programmatic connections;
• the nature of the connections is not exposed in text.

As it stands, using a chart generated from mermaid-js almost guarantees the following WCAG violations.

• 1.1.1 Non-Text Content, which the current text output fails to address;
• 1.3.2 Meaningful Sequence, as outlined in my comment here;
• 1.4.3 Contrast (Minimum), and I have made no effort to look for an existing issue,
• 4.1.2 Name, Role, Value, partially being addressed with More Accessible Mermaid Charts #2732.

I suspect 1.4.11 Non-Text Contrast, 1.4.12 Text Spacing, and 3.1.2 Language of Parts are at risk in some implementations as well.

It looks like this bug was logged in 2019 as #1013 and closed with no PRs or other visible code changes. It never should have been closed.

Microsoft's Fluid team attempted to address some of this in a PR: Add accessibility info to summary diagrams #3850, but that only addressed a flat structure. The example I embedded would not work with that structure.

If it helps, this is the alternative text I wrote for the chart above:

The decision tree: Using a flow chart in GitHub → Must it be accessible? If no: Wrong, try again. If yes → Into what format does it render? If JPG/PNG/GIF: Does it have an alt? If no: Wrong, try again. If Yes, and it is lengthy: Wrong, try again. If yes: Manually add a plain text description of the chart. Back up to Into what format does it render? If SVG → What role does it have? If none: Wrong, try again. If table: Wrong, try again. If grid: Wrong, try again. If presentation: Manually add a plain text description of the chart. If image → Does it have an aria-label? If no: Wrong, try again. If Yes, and it is lengthy: Wrong, try again. If yes: Manually add a plain text description of the chart. Back up to What role does it have? If tree → Does it contain any treeitem roles for the text nodes? If no: Wrong, try again. If yes → Are they contained in / owned by group roles beyond the first level? If no: Wrong, try again. If yes → Have you tested in any screen reader? Have you tested in any screen reader? If no: Wrong, try again. If yes → Did it accurately convey the relationships and structure of the chart? If no: Wrong, try again. If yes → You should write a CSUN talk about it.

As you can see, that is a bit verbose for a plain text description and lacks the nuance and structure you could represent with things like nested lists and clear descriptions of flow and relationship.

[brennanyoung]

While I strongly support this effort, the fact is that ARIA does not (yet) cover this kind of content very well. Still, there is much that could be added from today's ARIA which would improve the accessibility of mermaid content.

ARIA roles and attributes such as aria-label and aria-describedby are permitted on SVG sub elements (such as <g> or <text>). This is well supported by all major browsers and ATs.

We have the graphics-aria spec, which is a bare minimum for data viz. It offers three nameable roles

• "graphics-document"
• "graphics-object"
• "graphics-symbol"

I do wonder: Why is mermaid not using this spec already?

Another ARIA role I'd like to see used a whole lot more in mermaid is "group". Unlike the generic roles provided by default for <g> and <div>, it permits naming. This is a very nice and easy way to group content (for example a set of related datapoints), which is important in dataviz.

Adding a meaningful semantic role such as "group" or "graphics-object" will get that element added to the accessibility tree. The names of these roles are very general, so naming will help manage expectations.

For example, there (currently) is no ARIA role for "legend" but you can always do something like this:
<g role="group" aria-label="legend">
...or (using aria-roledescription)...
<g role="group" aria-roledescription="legend">

Datapoints may be associated with legend elements using aria-describedby.

I'd like to add that there is a discussion on the ARIA board about extending the graphics ARIA spec, to accommodate a wider set of dataviz 'primitives' - a proper ARIA spec for dataviz or infographic presentation.

For example, we would like to see explicit semantics for Axis, Axis value, Datapoint, Legend and (most important of all) a reliable way to connect one element to another, independent of the DOM order (e.g. in a flowchart).

I dearly hope that some of the folks here will join that discussion, contribute ideas, and perhaps gather insights which might be applied to mermaid.

w3c/aria#991

[weedySeaDragon]

@amitkumariiit @aardrian @brennanyoung Can we start a discussion on this, perhaps in Discussions:Ideas ?
Then all of the (great and pertinent) information you all have written won't be buried in this issue. (I'd start it, but feel like I'm the only one starting idea discussions and want to encourage others to do it, too.)

I'd really like to work on improving accessibility in mermaid. (I'll start with #3626 as soon as PR #3684 is merged.) I think that more good, tangible ideas and goals can be developed.

[weedySeaDragon]

meanwhile (unless/until this is moved into a discussion)....

Given the example flowchart, how could we show the possible sequence?
It seems that aria-flow is intended for this purpose but is not well supported. (Is that correct?)

Could aria-owns be used?

In a simple example: How should the sequence for a choice and the options be represented? ex:

flowchart TD
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]

Loading

I think that if we can come up with some ideas for aria labels and sequencing (order), I can implement some ideas so we can see how they play out.

[brennanyoung]

Hi there,

Yes, if aria-flowto behaved according to spec, this would be a solved problem. The spec permits multiple space-separated IDs to be included in aria-flowto. Seems ideal for a flowchart choicebox!

Unfortunately it doesn't really do anything in any assistive tech except JAWS, and (worse still) the ARIA folks are threatening to deprecate the attribute because of poor usage stats.

I am dismayed by this, because the real reason for those low stats is not that we don't need this feature, but that the spec is missing some important details that would make any implementation possible - as a result there's hardly any implementation in the wild.

For example aria-flowto does not suggest how the various flow destinations should be represented to the user. It's a list of IDs. OK, so what Ids? Are they the labelled branches ("One", "Two", "Three"), or are they the destination "boxes". ARIA has no role for "labelled branch" or "connector" or anything like that. (Yes, we do need this).

And then, what do we do with those IDs? Pull out the text nodes contained within and.. then how should they be represented by assistive tech?

After that, what if we want to offer an affordance for 'returning' having flowed somewhere (e.g. the visit-footnote-and-return use case). And what if multiple arrows point to the same box? The possibility of one-to-many relationships complicate the concept of "back" . Presumably the browser or the assistive tech needs to maintain some kind of 'history' stack? No such expectations have been mapped out. If they had, we might have aria-flowto working today. (It's not too late IMO).

The other thing that worries me is that the suggested replacement, aria-details is not an obvious choice for flowcharts and other directed graphs, and is not finalised yet. I invite all interested parties to examine the draft spec as is to determine whether this will be a viable mechanism for marking up flowcharts, sequence diagrams and the like. To my mind, the current draft spec is far too woolly for it to be an obvious choice for directed graphs.

That said, if aria-details turns out to handle this case (and, crucially, if there are working implementations with screen reader support), I will be delighted. So far though I am very skeptical about it.

aria-owns is not appropriate for this. All it does is set up a pseudo parent-child relationship in the accessibility tree, regardless of the relationship between those elements in the DOM. It's also poorly supported on Apple's platforms.

If there is anyone here that believes that aria-flowto has some value for marking up data representations over and above what is promised for aria-details (or even a contrary opinion) please contribute to discussion issue 630 on the ARIA board. This would be enormously helpful. Mentioning mermaid or other libraries that you may have used would not be out of place.

So, in general it is very difficult to offer "branching browsing affordances" for anything that goes against the standard DOM tree. As a result, some accessible dataviz experts such as Doug Schepers routinely use tab index as an affordance for data points, not because those points are supposed to be interactive, but because it's the only way to steer the AT flow from front end code. There is nothing corresponding to document.activeElement for the screen reader's reading caret, no focus or blur events, no option for keyboard handling.

For now, the only other thing in the ARIA spec that comes anywhere near a directed graph connector is [link](https://www.w3.org/TR/wai-aria-1.2/#link) (implemented in both HTML and SVG as <a>).

A flowchart choicebox might be imagined as a group of links, where the "link text" is derived from the label of the connector, so you might have something like this (rough pseudocode without any stroke/fill data):

<svg aria-roledescription="flowchart" height="160" viewBox="0 0 160 160" version="1.1" width="160">
    <title>Context for our fabulous product</title>
    <g role="graphics-object" aria-roledescription="terminal" id="start">
        <text>B</text>
    </g>
    <g role="group" aria-roledescription="decision" aria-labelledby="choicebox1">
        <text id="choicebox1">Let Me Think</text>
<!--We imagine the visual connector lines as "purely presentational", and expose the connections to a11y tree thus:-->
        <a href="lap">One</a>
        <a href="pho">Two</a>
        <a href="car">Three</a>
    </g>
    <g role="graphics-object" aria-roledescription="process" id="lap">
        <text>Laptop</text>
    </g>
    <g role="graphics-object" aria-roledescription="process" id="pho">
        <text>iPhone</text>
    </g>
    <g role="graphics-object" aria-roledescription="process" id="car">
        <text>Car</text>
    </g>
</svg>

(Flowchart nomenclature borrowed from https://en.wikipedia.org/wiki/Flowchart#Common_symbols )

BTW I support the idea of a broader discussion about a11y in mermaid, not all piggybacking on this one issue.

[weedySeaDragon]

I went ahead and created a discussion: #3689 .

Thanks for your really complete and informative posts about all of this @brennanyoung . I'm going to try to put something together. I feel like I need to think about this more and do a bit more reading before commenting on the aria standard. (Though I am going to try to head towards doing that.)

[mgifford]

Looking at https://mermaid.js.org/config/accessibility.html

Looks like SVG title and description are supported, which is good. However, more importantly would be critical to get the actual substance of the SVGs to be accessible.