Choosing an apt threshold for Visual Regression tests

[sharath-sriram]

Current Scenario

All existing components have a visual regression threshold of 0.1. This means that Visual Regression will not throw errors until the difference in the images crosses 0.1. While this works fine enough at the moment, there are times when the threshold fails to identify certain breaking changes for a few components. At such times, it is left to the developer committing the changes to responsibly identify the scope of error and reduce threshold to create new reference images and revert back to the old threshold.

What needs to be done ?

Given how each component has its own set of renders for which the visual regression tests are run, it will make sense for the developer to identify what might be an apt threshold for each of the components and set the same, ensuring that it doesn't fail on CI. It is definitely not a necessity that we should maintain a uniform threshold (open to arguments on this) for all components.

For more understanding on how the mismatch threshold is calculated, visit:
https://github.com/rsmbl/Resemble.js/blob/af57cb2f4edfbe718d24b350b2be1d956b764298/resemble.js#L495