04 Screenshots
Screenshots are crucial to the workflow advocated by this set of frontend testing tools.
They are what makes these testing tools frontend testing tools and not just any testing tools.
In this chapter we'll focus on both the philosophy behind them and the properties of the concrete screenshot mechanism included within this set of tools.
So, while working on a feature you'll most probably do some if not all of the following:
- write some application code (think React components)
- play with the app in the browser
- mock some APIs so that you don't have to seed and safeguard the data available in some staging environment
- develop test scenarios navigating the app (click here, click there, type that, etc.)
- make some assertions in these test scenarios
Let us focus on the assertions.
You might have already seen something like this:
expect(add(2, 3)).toEqual(5);It is a test that states that 2 plus 3 equals 5. It runs the system under test, which in this case is the add function and then makes sure the result is what we expect, that is 5.
It's so easy to write because it's not a big deal to come up with the expected result of adding two numbers while we're writing the test. By the time we write expect(add(2, 3)).toEqual( we already know it's 5 and we can just type it in.
But what if there's no easy to way to quickly state what is the precise result that we're expecting? What if there are so many data points that it's unreasonable to think that we'll list all of them in our test case, even if we care about all of them? Sometimes it's much easier to simply run the code and see if the result is correct. We can do exactly that. We start by running the code:
> add(2, 3)
5
Then we look at the result and judge whether it's correct. It seems all right, so we copy the result we got and paste it into the test case as the result we expect from now on.
You may ask: What's the point of such a test? The benefits are twofold.
Firstly, we did run the code and verified the results at least once when writing the test.
Secondly, we automated this verification process. So if the result unexpectedly changes in the future, we'll know about.
To sum it up, even though we might have not know precisely what was the expected result of running our code before we actually ran it, we were still able to run it and create an automated regression test.
In fact, this technique is so common that it's been built into Jest. Jest can remember what was the result of running your code the first time you ran it (when you were able to verify the output just by looking at it) and then raise an error if it ever changes. This is how you do it:
expect(add(2, 3)).toMatchSnapshot();The output will be recorded in a separate file and you'll be notified if the add function ever starts returning something else.
As you can see, snapshot testing opens a whole new world of possibilities when it comes to example-based testing. Now you can automatically test some examples of behavior even if it's not easy to precisely describe the expected result beforehand. One thing worth keeping in mind though is that whether they add value to your development process or not depends on whether you're able to quickly judge the recorded results. And that was what went sidewise in the past.
The application of snapshots in frontend tests started with things like "DOM Snapshots". For example, a snapshot like this:
expect(blueButton).toMatchSnapshot();would pass as long as it produced the following HTML code:
<button style="background-color: blue">Blue</button>It seemed all right at the first glance, but it turned out to cause more problems than in solved. The tests would still pass, even if some global style overrode the style of all button elements. In the same time, the test could fail after a perfectly fine refactor such as extracting this style to a CSS class. They slowed down development and didn't prevent regression, what means they failed to deliver on their promises.
But why did this happen?
Most of the time as an application developer you write code that in one way or another performs some actions. In the case of functions performing mathematical operations it's comptuing some results and in the world of frontend apps it may be rendering certain things on the screen, taking keyboard input, reacting to the mouse position etc. That's why this first snapshot test:
expect(add(2, 3)).toMatchSnapshot();recording a snapshot like this:
5
is something drastically differnet from this second snapshot test:
expect(blueButton).toMatchSnapshot();recording a snapshot like this:
<button style="background-color: blue">Blue</button>The first test monitors the expected result, which is simply the number 5. The second test monitors an intermediate value, which is some HTML code. We hope that when run it gives expected result, which is a blue button, but we don't really verify that part. In other words we could say that while the first test is a test of an end-user product (a mathematical operation that gives useful results), the other one tests a transpiler (a piece of code that produces another piece of code). And unless we're focused on delivering exactly that - a transpiler - there are other things we may want to tests, other things that bring value to our customers.
The key to benefit from snapshots if you're a frontend dev lies in using them to test the important output - how the app actually looks.
Jest alone is unable to provide that, but when used together with other great tools, such as Mapbox's pixelmatch and Selenium WebDriver (which is a W3C Recommendation) it becomes possible to get back on track with snapshots.
This distribution of frontend testing tools combines all of these tools and even more, so that you don't have to spend months and thus thousands of dollars fine-tuning them yourself.
You can get started immediately.
Just add screenshotTake("someName") in your tests:
test(
NAME,
() =>
testCtx[0]
.browserOpen()
// ...
.tlFindByText("form", "add")
.clickIt()
.screenshotTake("adding")
// ...
);When you run this test, it'll take screenshots:
And when app's appearance changes, you'll be notified like this:
If this looks fine to you, you can tell Jest to update the snapshot, just like that:
npm t happy_path -- -u
And if it's something you didn't expect, you are now aware that you need to fix it.
In life we face trade-offs. Screenshot testing is no different. When composing and configuring this distribution of frontend testing tools, I focused on making the developer experience as pleasant as possible, so that developers can qucikly develop tests and then maintain them and not feel dreadful.
The browser is exactly the same browser you have installed on your computer. It means you can interact with it without any friction, even offline, and there's absolutely no lag. You can use devtools and interact with the app like you always do.
These tools are built to provide you - the person writing tests - a delightful experience.
But that comes at a cost. Long story short, CI environments or teams where everybody uses a different kind of laptop may pose some challenges.
The screenshots are recorded locally. If you work on a Linux laptop and your CI environment is a similar Linux environment, then the screenshots should match and it should work just fine. On the other hand, if you develop on a Mac or a Window laptop and then run tests on Linux, it will cause problems due to the differences in rendering between these platforms.
What can you do about it?
You can set up your CI environment on a machine similar to the machine of at least one developer. For example, if there's at least one person working on a Mac, you can use another Mac to run your tests after each commit.
Another solution worth considering is sacrificing frontend tests on CI entirely. In the end, these are completely isolated tests that react to nothing but changes to the code. These are not smoke tests monitoring the production environment. Do you need them to run automatically on some centralized server? It'd be nice to have, for sure, but I believe such a possibility is not worth sacrificing the best possible developer experience that stems from using a local web browser.
If you're on a team and you use a Mac, another colleague runs Linux and yet another one has a Windows machine, then all three of you face a problem similar to the one described in the section about CI environments - the screenshots recorded on each of your machines will differ. You can still benefit greatly from these tools. You can still develop your client-side app even if the server-side is not ready yet, because the API mocking part is not affected. You can even still catch visual regression, because it'll work just fine on your machine for the screenshots you recorded.
But in such an environemnt you have to be more careful and disciplined, because your colleagues will rely on you even more and you'll rely on them more as well.
The rule for working in environments where everbody has a different dev machine is: please don't push changes that break something and remember to run the tests every time you pull changes made by your colleagues.
Frontend testing tools will use separate directories for screenshots recorded by Mac users, separate directories for Linux users and separate directories for Windows users. There will be no colisions there. But it also means we cannot expect the other person to update the screenshots for us if they have a different laptop.
Let's imagine you're a Mac users and I run Linux. You make some changes to the app and updaed the image snapshots after approving them. It's all good. But then when I run the tests right after pulling your changes they fail on my machine, because the Linux snapshots are outdated. What I do before introducing any changes to the code is I update the snapshots on my machine. From then on I can proceed with my work and enjoy all the benefits of screenshot testing. Finally, when my work is finished and I'm satisfied with the results, I push my changes. You repeat my workflow, that is you run the tests and if necessary you update your version of the snapshots before you change the code and we're all good. As you can see, the success of this approach depends on the commitment to pushing only working code. If I push some broken components, you most probably won't notice that you've just updated your snapshots to expect that component to be broken.
To sum it up: please either use similar laptops or never push broken tests.