Use arrow keys to drag balloon with keyboard

[jessegreenberg]

Should we add the ability to drag the balloon with the arrow keys when the screen reader is not in use? When the screen reader is turned off, we have full control of the entire keyboard.

@emily-phet and @jessegreenberg like this idea, but it would be good to check with the rest of the team. In the past, we have discussed that it could create an inconsistent experience if a blind student is working with sighted student. Confusion might arise if one interaction pattern works on one platform but not another.

[jessegreenberg]

One idea was to try to detect when a screen reader is in use and set the interaction keys accordingly. After some searching, it looks like this detecting screen reader use is not possible. AT do not provide a user agent string, they work on top of the browser. But all web discussion I could find about this is ~4 years old, so it would be good to verify.

Another option would be to enable the arrow keys and the WASD keys always. When supported arrow keys would work intuitively. When not supported, the descriptions would guide the user into using the WASD keys.

[jessegreenberg]

@jobara and @jhung agreed that it was not possible. For privacy reasons, it was decided in the community that the content creators should not be aware of a user's disability.

In 11/15/16 we decided that we will enable both arrow keys and WASD keys. As part of this we will have to determine how screen readers handle this, and determine how this impacts the navigation cues in the sim.

[jobara]

@jessegreenberg here's an interesting read on the subject https://www.marcozehe.de/2014/02/27/why-screen-reader-detection-on-the-web-is-a-bad-thing/

[jessegreenberg]

That is an excellent article @jobara, thanks very much! That all makes a lot of sense.

[terracoda]

@jessegreenberg, @jobara, @jhung, @emily-phet, I agree that it would be useful to enable the arrow keys. It is my understanding that with the role application that screen readers should be releasing the Arrow keys to the web application anyways. It is possibly a bug that they do not all do that right now.

@jessegreenberg reported in another issue that NVDA (with Firefox) did actually release the Arrow keys, but Jaws did not. If we enable the Arrow keys we can do some more testing with the latest browsers and screen readers.

@jessegreenberg, do you have any tips on finding old issues. I found this somewhat related old closed issue about the Keyboard Help #105.

[terracoda]

@jobara, thanks for posting Marco's article. It is very interesting and a very important consideration. We shouldn't need to know if a person is using AT to make a good experience for them. It sure would be nice if all the vendors implemented standards in more similar ways, however :-)

[jessegreenberg]

From https://www.w3.org/TR/wai-aria/roles#application

In my opinion, the role is defined pretty loosely. It seems like allowing arrow keys with role=application is sort of up to the AT's choice. Here is a rundown of the latest testing with role=application and the arrow keys:

• NVDA + Firefox: Arrow keys work
• NVDA + Anything else: role=application doesn't work unless user explicitly enters 'forms mode'
• JAWS + IE: Arrow keys sort of work, but events are mangled such that they do not support drag and drop. (This might be a bug, but could have something to do with "Where appropriate, assistive technologies that typically intercept other standard device input events, such as touch screen input, could switch to an application browsing mode that passes some or all of those events through to the web application")
• JAWS + Anything else: I think 'insert + 3' trick works
• VoiceOver: Arrow keys do not work with role=applciation, they seem to be used by the virtual cursor always.

do you have any tips on finding old issues.

@terracoda the only think I know of is to use github's search feature on the issues page. You can search through open and closed issues by keyword or number.

[terracoda]

@jessegreenberg, I think the definition for role application in the new WAI-ARIA 1.1 fits our situation better, and is a little clearer. However, I do agree that the language is still vague. It says:

assistive technology that intercept standard input events should switch to a mode that passes most or all standard input events through to the web application.

While reviewing the application role definition, I was reminded of some new attributes coming in 1.1 that might be helpful for us:

New properties aria-roledescription, aria-keyshortcuts, aria-placeholder, aria-current , aria-modal.

I am especially interested in aria-roledescription. I am thinking we can use that to replace the screen reader utterance, "Application" with something more meaningful like, "Draggable Yellow Balloon", or "Grabbed Yellow Balloon."

@jessegreenberg, @jobara would it be a good idea to start an issue for discussing the usefulness and applicability of new aria things?

[jessegreenberg]

@terracoda yes, that does sound useful! I think the new repo https://github.com/phetsims/a11y-research/ would be a perfect place for that kind of issue.

[terracoda]

@jessegreenberg, thanks for the repo cue!
I opened, Issue #6

[emily-phet]

@jessegreenberg - Just want to check in on this - if you're not already, can you go ahead and implement arrow keys for balloon dragging in BASE?

[jessegreenberg]

Yes, @emily-phet the commit above adds the arrow keys. Ill deploy a dev version with these added.

[jessegreenberg]

@terracoda @emily-phet this version has the arrow keys enabled.
http://www.colorado.edu/physics/phet/dev/html/balloons-and-static-electricity/1.1.5-dev.2/balloons-and-static-electricity_en.html

I will test how having both enabled in the sim works with various AT combinations in this issue.

[jessegreenberg]

The arrow keys also need to trigger the aria-live alerts. The above link needs to be modified for this.

[jessegreenberg]

Testing results:

• NVDA + Chrome
  • WASD keys work!
  • Arrow keys work!
• NVDA + Firefox
  • WASD keys work!
  • Arrow keys work!
• NVDA + IE11
  • WASD keys work!
  • Arrow keys work!
• JAWS + Chrome
  • WASD keys work!
  • Arrow keys do not work - occasionally the balloon will move slightly, presumably because of the strange way the AT sends the event to the browser
• JAWS + Firefox
  • WASD keys work!
  • Arrow keys do not work (same as Chrome)
• JAWS + IE11
  • WASD keys do not work.
  • Arrow keys do not work.

Unable to test VoiceOver for now, but I think we have consistently seen the arrow keys not work on that platform. Happy to see everything working on NVDA! And strange to see that nothing works on JAWS with IE11...

[jessegreenberg]

In all cases of JAWS, the arrow keys were obviously not working, so hopefully a user would be reminded to use the WASD keys. But it would be best to prevent all arrow key events on that platform. If I recall, the arrow key events are a constant stream of alternating keyup and keydown events for as long as an arrow key is pressed (this is why the balloon doesn't move at all). So it might be possible to detect when the user is trying to drag the balloon with the arrow keys with JAWS.

[terracoda]

@jessegreenberg, I did a very quick test on my MacBook Air

• macOS Sierra + ChromeVox
  • WASD works.
  • Arrow keys move the balloon, but with no alerts.
• My MacOS Sierra, VO + Safari
  • WASD move the balloon, but I get no alerts (this might be my system)
  • Arrow keys - slow movement left and right. Good movement up and down. Good movement all directions with added shift key. No alerts and issues with other descriptions getting out of synch.

[jobara]

@jessegreenberg and @terracoda I tried this quickly with Safari and VoiceOver on macOS Sierra. I did get the alerts ( direction, charges, etc. ) with the WASD keys. For the arrow keys, they just send the balloon flying off in the direction pushed, unless the shift key is used. With the shift key the movement seemed good, but as @terracoda mentioned, there were no VoiceOver updates. Also without the shift key the arrow also moves away from being on the button and performs the typical VoiceOver navigation. However the button is still grabbed, so if you go back you can arrow around more.

[jhung]

@jessegreenberg I just did a quick test with keyboard + Safari. My results are a little different from @terracoda's.

Mac OS Sierra, Safari, no VO

• After pressing key, the balloon moves and focus is removed from the balloon. You need to shift+Tab to put focus back on.
• I can only make one movement with arrow keys before the balloon becomes unresponsive to any keyboard input. i.e. Tab to focus balloon. Enter to grab. Right arrow to move balloon. Now focus is lost on the balloon (see the first point above). I press shift + TAB to put focus back on, and any arrow keys don't seem to do anything.

[terracoda]

@jhung, are you pressing Enter or Space to grab the balloon first? The pink outline should change to black.

@jessegreenberg, for me the balloon also went flying really far once when I pressed the left arrow key. Note that I did not try it without a screen reader yet.

Thanks @jhung for testing that!

The lack of alerts with my MacOS Sierra and VO seems to be an anomaly of my system.

[jhung]

@terracoda Yep. The behaviour is what @jobara describes (the balloon goes flying off in a direction). Putting focus back on to the balloon after that and attempting to move using arrow keys doesn't do anything.

Another funny thing is that once the balloon is in this unresponsive arrow state, trying to drag and drop with mouse has the balloon constantly moving back in the direction of the initial arrow key press... almost as if the key-up was never registered.

To reproduce:

• Tab to focus balloon. Press enter to grab.
• Press right arrow - balloon goes shooting off to the wall. Balloon would now lose focus.
• Shift-Tab back to the balloon.
• Press Enter to grab the balloon.
• Press any arrow keys - notice the balloon doesn't do anything.
• Use mouse and try dragging the balloon left-ward away from the wall. Notice how the balloon would move right even though you're moving left with the mouse (like you're fighting the right arrow keys of a ghost user at the keyboard).

[terracoda]

@jhung, very interesting!

[jessegreenberg]

@terracoda @jhung @jobara thanks very much for testing!

@terracoda regarding

Arrow keys move the balloon, but with no alerts.

This should be fixed with eb9e0a7. I will deploy a new version soon with this fix.

For the issue where the balloon flies off in a direction, I agree with @jhung, it sounds like the keyup event is never getting registered. @jhung @jobara were you by any chance testing with a bluetooth keyboard? This is exactly what we see when a bluetooth keyboard is used.

[jessegreenberg]

Testing with VO. I observed something different still:

• El Capitan (10.11.6) with VO:
  • WASD keys work
  • Arrow keys do not drag the balloon, but navigate the virtual cursor. If the balloon is picked up, I can still press the left and right arrow keys to find information about the balloon without releasing it.
• El Capitan (10.11.6) without VO
  • WASD keys work
  • Arrow keys work
• Yosemite with VO:
  • WASD keys work
  • Arrow keys both move the balloon AND navigate the virtual cursor! Once the virtual cursor leaves the balloon, arrow keys work as usual with the virtual cursor.
• Yosemite without VO:
  • WASD keys work
  • arrow keys work

Our sierra machine is out of office now, ill see if I can ping the borrower.

[emily-phet]

@jessegreenberg
I just took a quick look - the arrow keys don't work for me at all on Safari or Chrome with VoiceOver on. After grabbing the balloon, pressing the arrow keys moves the virtual cursor with no Balloon motion at all.

The arrow keys seem to work fine with VoiceOver off on Safari and Chrome.

[jessegreenberg]

Thanks @emily-phet. Do you also use macOS Sierra?

[emily-phet]

@jessegreenberg Yes, macOS Sierra, v10.12.1

[jessegreenberg]

Ok, thanks, good to know. I just tried on Yosemite 10.10.5 on Safari and Chrome with VO turned on and the arrow keys work there.

[jessegreenberg]

I also just tested on OSX 10.9 with VO in Chrome and Safari, arrow keys worked there too. I dragged the balloon for a while and the arrow keys continued to work when all charges were removed from the sweater.

[emily-phet]

Sierra broke it? Too bad!

[jessegreenberg]

Possibly! Though it is working on Sierra for me (comment #205 (comment)).

[terracoda]

@jessegreenberg, can you take a screen shot of your VO settings?

[jessegreenberg]

@terracoda this enough info?

[emily-phet]

Here's mine - looks like there's a difference for the "Navigate Images" option, "Always" vs "With Descriptions". Although, trying BASE with Safari and Chrome with the "Always" option (like Jesse's) results in same non-functioning arrow keys. :(

[terracoda]

Thanks! @jessegreenberg, @emily-phet I have tried both those settings. In the sims, the Navigate images setting doesn't seem to matter since there are no images in the PDOM.

[jobara]

@jessegreenberg I tried the arrow keys in the sim just now and they seemed to work as expected. I'm using macOS 10.12.2 and Safari 10.0.2. My settings looked like yours.

As a side note, I did notice a side effect of the way that the movement is calculated on keypress. If i tap rapidly, the balloon doesn't move at all. So it feels a bit broken. I think you could probably make it travel a minimum distance on keyup, this may also fix the JAWS issue.

[terracoda]

@jessegreenberg, @jobara, @emily-phet, ok, I just had the most surreal experience. I am using a MacBook Air Sierra 10.12.2. Since I was at Assets, the interactive alerts have never worked for me on this machine - no idea why. I had tried several VO settings trying to get it to work.

I just tried the sim with the arrow keys, and the interactive alerts (dragging, rubbing, and releasing) were all spoken. I accidentally used the Arrow key, then grabbed the balloon again and upon the second grab, the alerts were no longer spoken.

I reloaded and they were spoken again, and now they are spoken when using the WASD keys, too.

Note, just before this test I was just adjusting my VO verbosity settings because I was only getting "This item has a help tag" when grabbing the balloon. So I think the change in VO settings is what is the making this all work. I think I will make another issue to record my VO settings. This might be helpful for the future.

And the arrow keys work, but the left and right are problematic as described by @jobara

[jessegreenberg]

@jobara thanks for testing! Good to know that arrow keys were being captured by the application and not the virtual cursor.

If i tap rapidly, the balloon doesn't move at all. So it feels a bit broken. I think you could probably make it travel a minimum distance on keyup, this may also fix the JAWS issue.

This is a great suggestion, thanks, I created a new issue to track this because I think there are lots of things to investigate and discuss here.

[jessegreenberg]

@terracoda that is really interesting, let us know what you find in the settings!

[jessegreenberg]

Hmmm...I was just playing with the sim to test #216 and while using VO in Safari and the arrow keys suddenly stopped working for the balloon.

[jessegreenberg]

If I hold the shift key down, the arrow keyws work. Otherwise, they move VO's cursor.

[jessegreenberg]

@jobara, now that the arrow keys have stopped working in BASE, they have stopped working in other apps such as

http://build.fluidproject.org/infusion/demos/reorderer/imageReorderer/

And I can't get them to work for anything other than the cursor even when focus is on interactive content. Have you ever run into this before?

[terracoda]

@jessegreenberg, I just opened a new issue #217 which I found today. Everything stops working when I deplete the sweater of charges and have 2 balloons on the go.

[terracoda]

@jessegreenberg, in #205 (comment), I noted that the arrow keys stopped working after I depleted the sweater of charge. I just tried that again (with a single balloon), and I did not encounter a problem.

[terracoda]

@jessegreenberg, should I put my VO settings in a separate issue? Maybe others would like to compare and comment?

[jessegreenberg]

@jessegreenberg, should I put my VO settings in a separate issue? Maybe others would like to compare and comment?

Please do, that would be great!

[jobara]

@jessegreenberg when testing the sims earlier in the year i did run into cases where certain VO navigation just seemed to stop working. I reset my VO settings and things started to work again. This hasn't happened for a long time though, so I assumed it was just something that happened to my system. Not sure it's exactly the same as what you're seeing ( mine might have been related to tabbing ), but hopefully it will address your issue. It would also be interesting if you can consistently reproduce the issue.

To reset VO settings.

1. Open the VoiceOver Utility
2. In the menubar go to file > Reset All VoiceOver Preferences...
3. Click Reset

[terracoda]

@jobara, thanks for the VO reset tip. I haven't tried that. It would be good to compare all default settings with what I currently have set (phetsims/a11y-research#14).

[jessegreenberg]

Thanks @jobara I will give that a try next time I have access to an apple device.

[jessegreenberg]

Hmmm...I was just playing with the sim to test #216 and while using VO in Safari and the arrow keys suddenly stopped working for the balloon.

After work in john-travoltage, we now know why. VO has a "quick nav" setting that totally changes navigation strategies. The way to access this is to press left and right arrow keys at the same time. In "quick nav" mode, arrow keys cannot be used in an application.

I think this issue can be closed. We can use the arrow keys to move the balloon on all platforms except JAWS with Firefox, and a bug report has been submited to VFO. CLosing.