Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more info about a11y tests #76045

Merged
merged 7 commits into from
Sep 3, 2020
Merged
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,56 @@
[[development-accessibility-tests]]
== Automated Accessibility Testing

=== Writing Tests
myasonik marked this conversation as resolved.
Show resolved Hide resolved

To write an accessibility test, use the provided accessibility service `getService('a11y')`. Accessibility tests are fairly straightforward to write as https://github.com/dequelabs/axe-core[axe] does most of the heavy lifting. Navigate to the UI that you need to test, then call `testAppSnapshot();` from the service imported earlier to make sure axe finds no failures. Navigate through every portion of the UI for the best coverage.

An example test might look like this:
["source","js"]
----
export default function ({ getService, getPageObjects }) {
const { common, home } = getPageObjects(['common', 'home']);
const a11y = getService('a11y'); /* this is the wrapping service around axe */

describe('Kibana Home', () => {
before(async () => {
await common.navigateToApp('home'); /* navigates to the page we want to test */
});

it('Kibana Home view', async () => {
await retry.waitFor(
'home page visible',
async () => await testSubjects.exists('homeApp')
); /* confirm you're on the correct page and that it's loaded */
await a11y.testAppSnapshot(); /* this expects that there are no failures found by axe */
myasonik marked this conversation as resolved.
Show resolved Hide resolved
});

/**
* If these tests were added by our QA team, tests that fail that require significant app code
* changes to be fixed will be skipped with a corresponding issue label with more info
*/
// Skipped due to https://github.com/elastic/kibana/issues/99999
it.skip('all plugins view page meets a11y requirements', async () => {
await home.clickAllKibanaPlugins();
await a11y.testAppSnapshot();
});

/**
* Testing all the versions and different views of of a page is important to get good
* coverage. Things like empty states, different license levels, different permissions, and
* loaded data can all significantly change the UI which necessitates their own test.
*/
it('Add Kibana sample data page', async () => {
await common.navigateToUrl('home', '/tutorial_directory/sampleData', {
useActualUrl: true,
});
await a11y.testAppSnapshot();
});
});
}
----

=== Running tests
To run the tests locally:
myasonik marked this conversation as resolved.
Show resolved Hide resolved

[arabic]
Expand All @@ -20,4 +70,31 @@ The same thing that runs in CI, can be run locally using their browser
plugins:

* https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US[Chrome]
* https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox]
* https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox]

=== Anatomy of a failure

Failures can seem confusing if you've never seen one before. Below is a breakdown of what a failure might look like coming from CI:
myasonik marked this conversation as resolved.
Show resolved Hide resolved
["source"]
myasonik marked this conversation as resolved.
Show resolved Hide resolved
----
1) Dashboard
create dashboard button:

Error: a11y report:

VIOLATION
[aria-hidden-focus]: Ensures aria-hidden elements do not contain focusable elements
Help: https://dequeuniversity.com/rules/axe/3.5/aria-hidden-focus?application=axeAPI
Elements:
- #example
at Accessibility.testAxeReport (test/accessibility/services/a11y/a11y.ts:90:15)
at Accessibility.testAppSnapshot (test/accessibility/services/a11y/a11y.ts:58:18)
at process._tickCallback (internal/process/next_tick.js:68:7)
----

[arabic]
myasonik marked this conversation as resolved.
Show resolved Hide resolved
. The first two lines are, "Dashboard" and "create dashboard button" are, respectively, the names of the test suite and specific test that failed.
myasonik marked this conversation as resolved.
Show resolved Hide resolved
. Lower, always in brackets, "[aria-hidden-focus]" is the name of the axe rule that failed followed by a short description of it.
myasonik marked this conversation as resolved.
Show resolved Hide resolved
. On the next line, "Help: <url>", links to axe’s documentation on that rule. It usually includes info like severity, remediation tips, good and bad code examples, etc.
myasonik marked this conversation as resolved.
Show resolved Hide resolved
. The last relevant line is under "Elements:" which points you to where in the DOM the failure originated (using CSS selector syntax). In this example, the problem came from an element with the ID of example though this won’t be always as clear cut. If the selector is too complicated to find the source of the problem is, using the browser plugins mentioned above can help locate it. Otherwise, if you have a general idea where the issue is coming from, you can also try adding more unique IDs to the page to help you narrow down where the error is.
myasonik marked this conversation as resolved.
Show resolved Hide resolved
. Under the selector, is a stack trace which points to the internals of axe and is usually not helpful. It's there in case you believe the test failure to be a bug within axe and not in your code though this is unlikely.
myasonik marked this conversation as resolved.
Show resolved Hide resolved