Add visual regression testing

[36degrees]

What

Add visual regression testing to GOV.UK Frontend

Why

This came up as part of a 'tech & ops health check' exercise the team ran in November 2020, as part of the 'testing – pre-deployment' practise.

We scored ourselves as 'good' for that practise, but said that we were 'moderately concerned' about it as we know there are some areas that could be improved.

Visual regression testing was highlighted as one area we'd like to add to GOV.UK Frontend. We believe that it will help us to avoid shipping unintended changes to the design or layout of components, especially in non-default contexts (for example, changes that are only visible when using compatibility mode, or if the user is using Windows High Contrast Mode).

Who needs to know about this

Done when

• Trial Percy in GOV.UK Frontend repo - in progress until end of Q4
• Revisit how trial went and decide next steps

[vanitabarrett]

We're currently trialling Percy until the end of Q4. I'm going to move this into blocked until that trial has completed, and we can review how it's gone then and decide any next steps.

[oscarduignan]

Spotted the addition of Percy recently and wanted to share how we're doing our visual regression tests with the hmrc-frontend library in case it's a useful example when you revisit this after your trial of Percy because the libraries are so similar.

Rather than the HTML snapshots which Percy takes, we're just taking screenshots with a JavaScript library called BackstopJS. And, by default, we capture a mobile and desktop version of all the examples listed in each component's YAML definition.

We have a way to define alternate states (like focus states or having an interactive menu opened) and screenshot settings (like waiting before taking the screenshot) inline in the component/example YAML object. (described in our contributor docs)

We keep the screenshots in version control. So, we can use GitHub PRs to sign off the visual changes still. And GitHub rich diff support allows us to view before/after screenshots side by side, or overlaid with a slider / fader:

Then having a folder with screenshots of all the examples of all the components in the repo is quite useful for linking to examples when providing support / writing documentation.

[vanitabarrett]

Percy seems to be working pretty well for our team. People are either looking at the screenshots and finding them useful, or they haven't needed to do so yet - probably due to the nature of the things they've been working on.

Given we're only using ~43,000 screenshots per month in total (across the GDS account), we're not near our monthly limit so it makes sense to keep Percy running.

I've raised a new card for exploring other scenarios in which we may want to generate screenshots #2593 which we may want to pick up in future.