Support a11y on mobile devices #852
Comments
|
The above commit hides this behind a query parameter. In doing so, I noticed that a lot of this work has been spread around Display, AccessibilityTree, AccessibleInstance, AccessiblePeer...and attributes added to HTMLElements throughout these types are related and dependent. This will make maintenance difficult, it would be great if we could consolidate this better. |
|
I just noticed that the positioning isn't quite correct for some buttons in rutherford-scattering. But if the window size changes then it corrects itself. So maybe its not getting updated frequently enough. |
|
The above commit fixed #852 (comment). |
|
I think the last big question is how to make sure that the PDOM does not interfere with pointer input while it is on top of the simulation, while making sure that it works correctly with a screen reader. The most elegant way is In summary, we need I spent some time trying to find an alternative. I started by trying to see if I could make elements with VO touching while still using <!DOCTYPE html>
<html>
<style>
.graphics {
z-index: 1;
}
.backDiv {
min-width: 100vw;
min-height: 100vh;
background-color: rgba( 0,0,255,0.5);
position: absolute;
left: 0;
top: 0;
z-index: 2;
}
.frontButtons {
min-width: 14vh;
min-height: 14vh;
transform: rotate(0.5turn);
pointer-events: none;
}
.pdom {
position: relative;
z-index: 10;
background-color: rgba( 0,255,255,0.5);
}
.simDiv {
-webkit-user-drag: none;
user-select: none;
touch-action: none;
}
</style>
<head>
<meta charset="UTF-8">
<title>Page title</title>
</head>
<body>
<div id="simDiv" class='simDiv'>
<div class="pdom" id="pdom">
<!-- we want events on this to appear receivalbe, but we want all pointer events to reach the bottom one without ever going through this -->
<button id='frontButton' class="frontButtons">DOM Button</button>
<input type="button" id='secondButton' class="frontButtons">
<label for="secondButton">Second Button</label>
</div>
<div class="graphics">
<!-- we want the event to reach this without touching the top one -->
<div id="backDiv" class="backDiv">Maybe sim SVG</div>
</div>
</div>
</body>
</html>It is critical that This is a very simple example, and I haven't been able to get this working in the sim yet. Using Weinre with the sim, I also found that We are kind of reverse engineering VO here, I can't find any specification about this at all. This is a very atypical usage of HTML. So anything we do feels like a workaround and could be fragile in the long run. A totally different approach could be to use JavaScript somehow instead of |
That is not correct, I was able to prevent this from working again by adding When I remove those attributes in the sim, everything seems touchable, but it is a little hard to tell because everything is so out of place. |
|
Interesting, if I use |
|
In the simulation this seems to work for everything except the HomeScreenView and navigation bar! |
|
With the above commit it is working much better. I found that I also found that for focusable elements with labels, VoiceOver will include the label bounding client rectangle in its black highlight for the focusable node. This was a problem because VoiceOver will send a fake pointer event at the center of this rectangle. So if the label is shifted off screen, the event will be at the wrong location. I tried to overlap labels and focusable elements, but then the "touch-to-focus" feature broke again because touching the element only read the label. I tried many CSS like hidden, visibility, text fonts, opacity, others, but none of them worked. As a workaround, I am making the label element tiny, and putting it at the very bottom of the focusable element. It looks something like this: The tiny white dash below the check box is the label. This way, the label doesn't overlap with the input, but it is near enough to the input that VoiceOver will send a fake pointer event to the correct location. |
|
AccessibilityTree currently has
I don't think this is necessary anymore, if UPDATE: When I remove the recursive nature of updateCSSTransformsForSubTree, things no longer work correctly. Why? UPDDATE: I had to remove this line from // no need to continue searching down this sub tree after this traversal
rootInstance.peer.descendantTransformDirty = false;UPDATE: This works, but I am not actually sure if it is better yet. UPDATE: It isn't better, and the current approach of finding the subtree to update doesn't work if we remove this. We need to check the entire subtree under an AccessibleInstance marked as dirty so we can keep UPDATE:
This is just a bad idea, what if some root level element updates every frame? |
|
I just noticed in pdom-transfer branch that NumberPicker is not focusable. I just merged master into the branch and the problem is still there. What changed? |
|
At one point it looked like tabindex played a role in what was and wasn't touchable with VO, so it was removed. Hopefully we can add it back now. |
|
Hmm, yes tabIndex does seem to impact this, but I don't know why. Some buttons behave as though their touch ares are not around the highlight or VoiceOver highlight. This is not the case with Why does VO care about this? Isn't tabindex=0 the default for focusables? This may or may not be helpful: https://stackoverflow.com/questions/18167981/clickable-link-area-unexpectedly-smaller-after-css-transform |
|
It was as though the touch area was around the top left corner of the element, so I thought it could be related to these lines of the CSS domElement.style.top = '0px';
domElement.style.left = '0px';
domElement.style.padding = '0px';
domElement.style.transformOrigin = 'left top';According to https://developer.mozilla.org/en-US/docs/Web/CSS/top
So I changed AccessibilityUtil.createElement to if ( focusable ) {
domElement.style.position = 'fixed'; // helps move container elements out of the way, doesn't seem to interfere
}
else {
domElement.style.position = 'relative';
}and the touch areas are much improved. This is too crude, and won't work for label siblings. We need to update these accordingly when focusable changes, but this is a very helpful find. |
|
#852 (comment) is not correct. I had accidentally left |
|
The following button has this problem: <button id="15-25-51-52-53-50" aria-roledescription="Sim Screen" style="transform-origin: left top 0px; pointer-events: none; transform: matrix(2.12098, 0, 0, 13.4891, 606.111, 367.688);">Plum Pudding Atom</button>If I remove EITHER |
|
What if we change to use width, height, and positioning instead of transform? The idea is we would calculate the right transform, then apply to the clientWidth and clientHeight (before any transforms), then use the transformed element to get the width and height from getBoundingClientRect(), then apply with |
|
Wow, this is actually working really well, this button behaves great in Safari: <button id="15-25-51-52-53-50" aria-roledescription="Sim Screen" style="position: fixed; pointer-events: none; width: 361.200; height: 374.535; top: 456.59; left: 259.61">Plum Pudding Atom</button>Where the width and height were calculated from getBoundingClientRect(). Ill try it in the sim. |
|
Oh my goodness this is working so well! |
|
I added unit tests for this in the branch. I am going to run through aqua tests once more then merge with master. |
|
aqua tests are running well so I am going to proceed with merging. Ill continue to watch CT for the next couple of hours though. @zepumph can we go over this issue at our next a11y dev meeting? I would like to check PDOM changes with you, but am also curious of your thoughts on the changes to AccessiblePeer. Like am I correctly adhering to model-view separation for a11y code in scenery, or should some things be moved out of AccessiblePeer/AccessibleInstance, and so on. |
|
The above review comments came from discussion with @zepumph, @jessegreenberg will address these review comments. |
|
In phetsims/a11y-research#133 it was discovered that NVDA sends fake pointer events (in addition to the change event) upon activation of check boxes. There are probably other cases/devices that do this too. Previously this wasn't a problem because the pointer events were well outside of the sim div. But now that elements are on top of the display they are getting caught. The result is that check boxes are activated twice, once on When this happens, the target of the DOM event is within the PDOM, so I am wondering if we can prevent touch events if they are sourced from PDOM elements. EDIT: This cannot be done in Input.js as the touch event has already come hit the input element in the PDOM before bubbling up to the root. EDIT: And to correct the above, it is the EDIT: In the following JSFiddle, we get all of these events when NVDA is turned on: EDIT: Adding these to AccessiblePeer seems to fix the problem: this._primarySibling.addEventListener( 'touchstart', function( event ) {
console.log( 'caught touch start' );
event.preventDefault();
event.stopPropagation();
} );
this._primarySibling.addEventListener( 'touchend', function( event ) {
console.log( 'caught touch end' );
event.preventDefault();
event.stopPropagation();
} );
this._primarySibling.addEventListener( 'mousedown', function( event ) {
console.log( 'caught mouse down' );
event.preventDefault();
event.stopPropagation();
} );
this._primarySibling.addEventListener( 'mouseup', function( event ) {
console.log( 'caught mouse ups' );
event.preventDefault();
event.stopPropagation();
} );With the above changes, we are still receiving an error because NVDA is trying to send the EDIT: OH, the label received the click event because I pressed enter while the virtual cursor had seleted the label element. |
|
#852 (comment) was fixed via #949. |
|
@jessegreenberg, what more needs to be done here? Is there anything I can do to help? |
|
This issue can be closed, we have been using this for a long time and it has been working well. This became sort of a mega-thread for issues related to mobile a11y. I reviewed comments in this issue and it looks like everything noted has been resolved. Other issues since then that are related to this have been opened, which is what we should continue to do in the future. |
This is a more up to date continuation of #41 which has gotten old and a bit stale. See that issue for documentation about our thoughts on the current approach for this and how it is working.
The text was updated successfully, but these errors were encountered: