[pa11y-ci] Use pa11y-ci-reporter-html instead of pa11y-reporter-html

[julien-deramond]

This PR is a proposal to help fix twbs@134dc7c based on the understanding I have for the moment of the reporters and the statuses of the several pa11y* projects. Hope it will help, otherwise this PR can be closed.

Links

• HTML report / artifact: https://github.com/julien-deramond/bootstrap/actions/runs/1584765955
• CLI output: https://github.com/julien-deramond/bootstrap/runs/4540114235?check_suite_focus=true

Summary

According to the pa11y-ci documentation it is possible to do:

npm install pa11y-reporter-html --save
pa11y-ci --reporter=pa11y-reporter-html https://pa11y.org/

Maybe I have missed or don't understand something but I tried it and nothing was generated nor displayed. So I suppose that if it doesn't work there, it is normal that it doesn't work neither with multiple reporters defined as:

{
    "defaults": {
        "reporters": [
            "cli",
            "pa11y-reporter-html",
        ]
    }
}

So I tried to understand what's going on here.

By adding some logs in node_modules/pa11y-reporter-html/lib/reporter.js, we can see that the HTML code is well generated. But it doesn't create a file containing this HTML code nor displays this HTML code as an output.

We can find another information in https://github.com/pa11y/pa11y-reporter-html that specifies that "The reporter has been merged into pa11y/reporters.".

Tried it and indeed, I was able to do this as well that produced exactly the same non-result:

{
    "defaults": {
        "reporters": [
            "cli",
            "./node_modules/pa11y/lib/reporters/html.js"
        ]
    }
}

Note: node_modules/pa11y/bin/pa11y.js -r html https://example.com displays the HTML as an output.

So let's dive into pa11y/reporters. They all create the content but never really create a file or display themselves their output.

Contrary to the reporter defined in pa11y-ci/reporters. At the end of the json.js, we can see:

if (fileName) {
  writeReport(fileName, jsonString);
} else {
  console.log(jsonString);
}

IMO, for a reporter to work with pa11y-ci, it must be capable of creating the file.

With all that in mind, I would say we don't have a choice for the moment and should generate a JSON file with ['json', { 'fileName': './pa11y-ci-results.json' }] and then use pa11y-ci-reporter-html to generate the HTML file from this JSON file 🤷