What should an a11y RC test look like

[KatieWoe]

As we remade the RC template (/issues/19) we noticed that the a11y section seems a bit sparse. It has, in the past, been included in the general RC and simply included making sure everything reads or functions correctly, nothing interferes with the sim, and looking at the "script" to see if anything is left out. I wanted to ask relevant parties if they feel this level of testing is sufficient for an RC test, and looking for suggestions and input for a more structured test.
See #19 (comment) for the new RC test template.

[emily-phet]

@KatieWoe Thanks for including us in this discussion! Before I comment further, it would be helpful for some clarification - forgive my ignorance of this - on how touch vs cursor-based interactions are tested? Is it just that on certain touch-specific devices the touch interface is tested, and on other devices a cursor-interaction is used?

[KatieWoe]

In general at the moment, on devices that support touch and cursor we try to test both, but we probably focus a bit more on cursor-interaction. On devices that only support one, we only test that one method.

[emily-phet]

I'm going to suggest a structure where the "General RC Test" category is expanded to include different specifics on different modalities - rather than having a "General" and an "A11y" category.

For example - in the first bullet of the current "General" category is "click every single button". There are lots of ways to click a button... mouse, touch, activating a button with the spacebar. You can get to each button in multiple ways - moving your cursor and possibly getting a hover state change, tapping with finger (no cursor movement, no hover state), moving focus with keyboard keypresses (getting moving focus highlight, and possibly a 'hover' state), etc.

What do you think about being explicit about mouse, touch, keyboard (no screen reader), and screen reader testing (each with a collapsible section), each possibly specifying devices/browser options or requirements, and each with a bullet list of testing steps. Then at the top of the testing request, it can be specified which set of these modalities need to be completed.

[KatieWoe]

The idea we had was that the developer making the request would delete whatever section is not included in the test. So, if there is no keyboard support the QA team wouldn't even see those instructions. This is meant to both cut down on the length of the final issue and prevent confusion as to whether something should be included or not.
I like the idea of making a list of the types of interactions to check on a device. Once a11y testing is happening with most RC tests I think we should combine it with general (same with phet-io) but until then I think the risk of miscommunication as to what should be tested can be avoided by putting it in a section that can be simply added or taken away.
In regards to the method of testing, it sounds like you would like some type of matrix that keeps track of methods of interaction on supported platforms.

[emily-phet]

@KatieWoe
Again - forgive me for asking a really basic question...

When a tester is doing RC testing, in what contexts/platforms/etc. are they doing the 'play with sim normally' and 'try to break the sim'. Are they doing this type of testing across all device/browser/etc listed? Or are they doing this rigorously in some environments, and then doing spot checking in others?

[KatieWoe]

They should do it fairly rigorously on all the devices they test on. Usually more time is spent on the new platforms, or heavy use platforms. A tester also will likely spend more time on this step on the first device they test. But, the process is still a bit more than a spot check for all the devices.

"Play with sim normally" means pretend they are a student and test the types of behaviors that would be expected. Things that are likely to occur.
"Try to break the sim" means test edge cases. Push thing to the extreme, beyond what would be expected by typical use.

[ghost]

Should we close this issue?

[KatieWoe]

I think so, unless @emily-phet and @terracoda have more to add.