Add pa11y-ci for automated accessibility tests with axe runner

[julien-deramond]

This Draft PR is originated from both twbs#29315 and Boosted use of pa11y / pa11y-ci

Run

Local

CLI mode

$ npm run docs accessibility-cli

  Your application is ready~! 🚀

  - Local:      http://localhost:9001
  - Network:    Add `--host` to expose

────────────────── LOGS ──────────────────

  [21:38:50] 200 ─ 3.85ms ─ /sitemap.xml
Running Pa11y on 137 URLs:
  [21:38:50] 200 ─ 4.32ms ─ /docs/5.1/about/overview/
  [21:38:50] 200 ─ 6.26ms ─ /docs/5.1/getting-started/accessibility/
  [21:38:50] 200 ─ 2.97ms ─ /docs/5.1/assets/css/docs.css
  [21:38:50] 200 ─ 3.45ms ─ /docs/5.1/dist/css/bootstrap.min.css
  [21:38:50] 200 ─ 1.41ms ─ /docs/5.1/assets/js/docs.min.js
  [21:38:50] 200 ─ 1.96ms ─ /docs/5.1/dist/js/bootstrap.bundle.min.js
 > http://localhost:9001/docs/5.1/about/overview/ - 8 errors
  [21:38:53] 200 ─ 1.25ms ─ /docs/5.1/components/accordion/
 > http://localhost:9001/docs/5.1/getting-started/accessibility/ - 8 errors
  [21:38:54] 200 ─ 0.71ms ─ /docs/5.1/examples/album/
 > http://localhost:9001/docs/5.1/examples/album/ - 0 errors
  [21:38:56] 200 ─ 1.12ms ─ /docs/5.1/components/alerts/
c > http://localhost:9001/docs/5.1/components/accordion/ - 8 errors
  [21:38:58] 200 ─ 0.64ms ─ /docs/5.1/extend/approach/

JSON mode

$ npm run docs-accessibility-json # Will generate `pa11y-ci-results.json`
$ npm run docs-pa11y-html

Focus on a specific page

diff --git a/build/.pa11yci.json b/build/.pa11yci.json
index a5b3d1e96..beb85877a 100644
--- a/build/.pa11yci.json
+++ b/build/.pa11yci.json
@@ -7,5 +7,8 @@
       "axe"
     ],
     "hideElements": ".bd-search"
-  }
+  },
+  "urls": [
+    "http://localhost:9001/docs/5.1/about/overview"
+  ]
 }
\ No newline at end of file
diff --git a/package.json b/package.json
index af3305af2..99b7e0733 100644
--- a/package.json
+++ b/package.json
@@ -35,8 +35,8 @@
     "css-prefix-main": "postcss --config build/postcss.config.js --replace \"dist/css/*.css\" \"!dist/css/*.rtl*.css\" \"!dist/css/*.min.css\"",
     "css-prefix-examples": "postcss --config build/postcss.config.js --replace \"site/content/**/*.css\"",
     "css-prefix-examples-rtl": "cross-env-shell NODE_ENV=RTL postcss --config build/postcss.config.js --dir \"site/content/docs/$npm_package_config_version_short/examples/\" --ext \".rtl.css\" --base \"site/content/docs/$npm_package_config_version_short/examples/\" \"site/content/docs/$npm_package_config_version_short/examples/{blog,carousel,dashboard,cheatsheet}/*.css\" \"!site/content/docs/$npm_package_config_version_short/examples/{blog,carousel,dashboard,cheatsheet}/*.rtl.css\"",
-    "docs-pa11y-cli": "delay 3 && pa11y-ci --config build/.pa11yci.json --sitemap http://localhost:9001/sitemap.xml --sitemap-find https://getbootstrap.com/ --sitemap-replace http://localhost:9001/",
-    "docs-pa11y-json": "delay 3 && pa11y-ci --config build/.pa11yci.json --sitemap http://localhost:9001/sitemap.xml --sitemap-find https://getbootstrap.com/ --sitemap-replace http://localhost:9001/ --json > pa11y-ci-results.json",
+    "docs-pa11y-cli": "delay 3 && pa11y-ci --config build/.pa11yci.json",
+    "docs-pa11y-json": "delay 3 && pa11y-ci --config build/.pa11yci.json --json > pa11y-ci-results.json",
     "docs-pa11y-html": "pa11y-ci-reporter-html",
     "docs-accessibility-cli": "npm-run-all --parallel --race docs-serve-only docs-pa11y-cli",
     "docs-accessibility-json": "npm-run-all --parallel --race docs-serve-only docs-pa11y-json",

GitHub CI

At the end, you can retrieve the HTML report:

Then, open the index.html file in the .zip file:

Topics to tackle

Find a way to use the HTML reporter but still log the results in CLI

It seems possible in the master branch of pa11y-ci via pa11y/pa11y-ci@5c842cf that allows multiple reporters.

We should be able to do something like:

{
    "defaults": {
        "reporters": [
            "cli",
            "pa11y-reporter-html" // or "json" maybe
        ]
    },

But we would have to wait for the 2.4.3 or 2.5.0 to be released.

For the moment, the CLI version is only available locally.

See if we can use npx maybe, since Chromium is a huge download or see if we can cache puppeteer on CI

XhmikosR: "I don't want the huge puppeteer (which in turn downloads Chromium) devDependency, ~143MB... Not sure how to work around this, we could probably move the needed packages in the Action that needs them, but it's not a perfect solution either"

Which runner?

• axe like Boosted?
• htmlcs like Bootstrap POC
• both?
• others if any?

axe runner: Analyze and fix the errors or use hideElements when necessary

Tests of the 20th of October

• "hideElements": ".bd-search": 1259 errors
• "hideElements": ".bd-search, footer ul.text-muted > li" (removes 3 contrast errors in the footer): 978 errors (-291 errors) — Could be fixed by changing the text color here