feat: Integrated Visual Regression with ember-backstop

[sharath-sriram]

Visual Regression for Nucleus Component Library using ember-backstop

Motive

A high performing component library like Nucleus needs to meet high development standards. One of the primary needs during the development of a Component Library is to ensure consistency with the Design System's Specifications during the development process. Human error can often lead to slight deviations in component look and feel which might go unnoticed by the developers. Visual Regression Testing solves this problem by automating the process of testing and identifying the differences in the appearances of components.

Why ember-backstop ?

ember-backstop is an open-source Visual Regression testing tool developed as an Ember Adddon using the foundations of the popular BackstopJS Visual Regression Testing Library.

ember-backstop promises visual regression using pixel by pixel diff comparison with parallelisation capabilities for performant testing. It works using Chrome Headless with Puppeteer as the underlying engine. Almost all of the features of BackstopJS such as advanced scenario handling for animations using hover and clicked selector mechanisms, setting customised threshold limits, customised delay before taking the screenshot are available for use directly in ember backstop for a case-by-case basis.

It is worthwhile to note that while ember-backstop does not provide sophisticated approval mechanisms over CI like ember-percy, it seamlessly works with locally approved reference images to perform the CI tests.

Architecture

As the Nucleus Component Library is a mono repo, we had to make deliberate choices on where to position our Visual Regression Tests. With the goal of ensuring highly independent packages within the mono repo, Visual Regression tests have been written inside each individual package and they run separately.

ember-backstop is installed as an addon for each individual package thus creating a backstop-remote for each of the packages. The reference images for the package are stored within the backstop folder within the package.

As multiple instances of backstop-remote cannot run at the same time, the tests for the packages have been serialised. Once the reference images are approved, any differences made to the components are caught by the tests. The changes can then be viewed by the report created by ember-backstop and further action can be taken.

Future Work

1. As the tests have been serialised, the run time on CI and local setup takes double the time compared to parallel tests. The performance of the tests can be improved by ensuring that tests are run only for those components whose dependent visual elements are changed.
2. Currently, any developer running the setup locally can approve the screenshots and make changes to the components. Going forward, keeping the open-source goals in mind, a workflow is required such that the reference images can be approved only by key maintainers.

[shibulijack-fd]

Add these to .gitignore

#backstopjs
html_report
bitmaps_test

[abhilashlr89]

@robin-thomas-e1391 @aslam-fw We need to get this into our engines repo. @shibulijack-fd @sharath-sriram Would be a great idea for you to post a talk/blog on this!

[sharath-sriram]

Sure, we are still ironing out some bugs. Once its functioning properly, we will write about it.