feat(ci): add test report check workflows (garris#1533)

[dgrebb]

Draft

---

Summary

This introduces more thorough GitHub testing workflows.

It toes the line of schema validation, but picks up some low-hanging fruit for any potentially breaking changes in PRs. These could be considered complimentary to the basic npm scripts already configered.

What are they Testing?

Perhaps a portion of the backstop test command was accidentally commented out:

module.exports = {
  execute: function (config) {
    const executeCommand = require('./index');
    // if (shouldRunDocker(config)) {
    //   return runDocker(config, 'test')
    //     .finally(() => {
    //       if (config.openReport && config.report && config.report.indexOf('browser') > -1) {
    //         executeCommand('_openReport', config);
    //       }
    //     });
    // } else {
    //   return createBitmaps(config, false).then(function () {
    //     // return executeCommand('_report', config);
    //   });
    // }
  }
};

Running npm run sanity-test does not catch this:

COMMAND | Executing core for "test"
COMMAND | Resolved already:test
COMMAND | Command "test" successfully executed in [0.001s]

Comparing the report.json does, as it is never generated:

![[Pasted image 20240115001855.png]]

Maybe someone changes a report property name:

![[Pasted image 20240114234903.png]]

Both are forced examples, but provide a glimpse into what is possible.

General Workflow Steps

Each workflow has the same basic structure, with caveats for smoke and integration:

1. Run an npm test
2. Compare the resultant report.json to a fixture of the corresponding test (./test/__fixtures__/[npm-script-name].json)
3. Before diff is run, filter report.json properties down to shape only
4. Pass/Fail and summarize results

Smoke Caveat

I've seen a few smoke tests pass on GitHub but fail locally. For now, this is solved by deleting the properties we know will have different shapes (or not exist at all in a pass):

jq 'walk(if type == "object" then with_entries(.value |= if type == "object" or type == "array" then . else "" end) else . end) | del(.tests[].pair.diff, .tests[].pair.diffImage)' test/__fixtures__/smoke-test.json

diffImage doesn't exist on passing tests, so it's removed before analyzing report.json . "misMatchThreshold" : 0.1, could also be bumped a bit, to be more forgiving.

![[Pasted image 20240115022206.png]]

Integration Caveat

The integration-test script generates two reports. One when running backstop reference, and the other after backstop test. We apply a fancy bash one-liner to find the most recently modified directory, and only diff the final report.

Details

First and foremost, .tests[.pair] object values are empty. Values will never be 1:1, due to system runtime differences, browser changes over time, etc. Data shape only is being tested in these workflows.

We use jq to traverse the report.json object and set non array or object property values to empty strings: "", and do the same for nested properties in any aforementioned object or array.

This allows us to test the general "shape" of data we expect backstop test to produce, comparing it with corresponding JSON files in test/__fixtures/.

That ends up looking like this, which is the shape tested in integration and sanity:

{
  "testSuite": "",
  "tests": [
    {
      "pair": {
        "reference": "",
        "test": "",
        "selector": "",
        "fileName": "",
        "label": "",
        "requireSameDimensions": "",
        "misMatchThreshold": "",
        "url": "",
        "referenceUrl": "",
        "expect": "",
        "viewportLabel": "",
        "diff": {
          "isSameDimensions": "",
          "dimensionDifference": {
            "width": "",
            "height": ""
          },
          "rawMisMatchPercentage": "",
          "misMatchPercentage": "",
          "analysisTime": ""
        },
        "diffImage": ""
      },
      "status": ""
    },
    {
      "pair": {
        "reference": "",
        "test": "",
        "selector": "",
        "fileName": "",
        "label": "",
        "requireSameDimensions": "",
        "misMatchThreshold": "",
        "url": "",
        "referenceUrl": "",
        "expect": "",
        "viewportLabel": "",
        "diff": {
          "isSameDimensions": "",
          "dimensionDifference": {
            "width": "",
            "height": ""
          },
          "rawMisMatchPercentage": "",
          "misMatchPercentage": "",
          "analysisTime": ""
        },
        "diffImage": ""
      },
      "status": ""
    }
  ],
  "id": ""
}

Happy to discuss in detail :)

Integration

Runs npm run integration-test, then tests the resultant report.json , which is the last step in the project's integration test: backstop test.

The GitHub workflow results in a pass/fail based on shape alone. The unfiltered A/B fixture/CI diff is included in the workflow's summary for further analysis.

![NOTE]
All workflow summaries contain the unfiltered diff. There will always be timestamp directory-name differences in the "test" property.

![[Pasted image 20240114214019.png]]

Workflows

Sanity

Runs both sanity-test and sanity-test-playwright then compares the corresponding fixture and report.json.

Smoke

Runs both smoke-test and smoke-test-playwright then compares the corresponding fixture and report.json.

Docker Sanity/Smoke

Same, but uses Docker.

Conclusion

A lot of options! Please let me know what you end up keeping and tossing. This is a level of testing I'm not sure the project needs, but would help identify PRs that introduce side-effects to the overall testing ecosystem.

Cheers!