Detect visual errors (visual verification testing) (OSOE-111)

[Piedone]

Detecting visual errors with something like Applitools, or generally detecting regressions and validation errors automatically somehow.

We could also use verification testing with Verify. Or perhaps Shouldly offers this too? Verify has support for Selenium. This may only compare HTML though (it depends on the use-case if a change in the HTML is an issue or not; sometimes we want to change the HTML without the visuals being affected, sometimes with just CSS changes the other way around). Verify.ImageSharp might be interesting for screenshots but seems to only compare metadata. Ideally, we'd offer a visual diff of screenshots.

APIs can use Verify.AspNetCore or Verify.Http too.

If using Verify or Shouldly this is what the workflow would look like:

1. Turn on visual verification testing globally or for a given test, or call an assertion method. I.e. the same how we do this for HTML validation and accessibility checks.
2. The first time you run the test you get an error and need to accept the output as the correct one.
3. Subsequently, you only get an error if the output changes. Then you have the option to again accept it if the change is expected, or fix the app.

Verification testing is not a silver bullet though: Verification testing is super simple to set up but if there's a mismatch in the approved and received output, it's sometimes hard to figure out what causes it (since it's a very high-level test). Manual assertions would still be necessary but visual verification testing can be run as another layer of checks.

[SimonCropp]

yes, Verify offers a visual diff of screenshots

[Piedone]

Ah, that's awesome, thanks for coming here.

[SimonCropp]

OOTB Verify does a binary comparison of images. Advanced comparison can be done by comparers, of which there are 3 image comparers

[Piedone]

Which of the image comparers would offer something along the lines of producing an image with the diffs being circled in?

[SimonCropp]

the comparers dont do that. the diff tool takes care of that. for example this is how it looks when using beyond compare

[Piedone]

I see, great, thank you!

[SimonCropp]

it is best demonstrated with a demo. let me know if you ever want to jump on a call and i can show you.

[Piedone]

I'd really appreciate that. I've sent you an e-mail to your address under your profile.

One thing to note here that this should work both in a local development environment and in CI. Shouldly does this by opening a diff tool locally (which is useful during development but wouldn't work in CI) and writing a diff to an exception message (and thus build output) in CI. I'd imagine something similar, in CI creating a diff image that we can save to the build artifacts, for example.

[Piedone]

We could do this in exactly the same way as HTML validation and accessibility checking: Optionally run on every page load, or explicitly with an assertion call.

The ImageSharp.Compare library looks like an interesting tool to do comparisons. We could do a simple approach like this too (though ported to ImageSharp probably) but we need tolerances because screenshots most possibly won't be perfectly the same.

Shouldly's MatchApproved() only supports text and thus not really suitable but we could use that to compare HTML (which might not be useless since we can snapshot the HTML after every client-side logic was run, so not just what was initially sent to the browser).

[Piedone]

Related Atata issues:

• Atata Screenshot Comparison atata-framework/atata#334
• Testing of screenshots.... atata-framework/atata#147