Increasing E2E performance and speeding up tests.

[Sidsector9]

Hello folks!

For a while, I have been experimenting with E2E tests for Restricted Site Access, trying to find ways to minimise the time it takes for the entire Cypress suite to finish.

Initial implementation

Here, the test application state was set up entirely by navigating through UI. The flow for each test was somewhat:

1. Navigate to plugin's admin settings page
2. Change settings through UI
3. Save settings which caused a reload
4. Navigate to the front/admin end where the actual assertions were run
5. Repeat 1-4 for the next test in the suite.

Problem:

The navigation between pages and the page reloads wasted a lot of time. In fact, even Cypress considers building up state using UI as an anti-pattern.

1st experiment to avoid UI

To achieve this, I made use of the custom cy.wpCli() command. What this does is runs WP-CLI under the hood.

Problem:

1. This wasn't as fast as I expected. Running WP-CLI commands directly through bash is fast, but inside @wordpress/env it would be: wp-env run tests-cli "wp option update key value". The issue here is the chain of commands: From Node > bash > PHP. This took way lot of time to execute within Cypress.
2. Setting complex application state. Some plugins store all the data under a single options key in a serialised format in the DB. This is not easily possible to do using WP-CLI as it requires the option values in JSON which will be converted to serialised format under the hood.

2nd experiment to avoid UI

@wordpress/env allows you to mount plugins using the .wp-env.json config file. In this approach I created a simple plugin what I call as a seeder plugin. The sole responsibility of this plugin is to "seed data" through rest routes, which are called just before running tests.

For example, if we have a test file for verifying plugin activation/deactivation as activation-deactivation.test.js, the seeder plugin registers a route wp-json/rsa/v1/seed/activation-deactivation to set state applicable for the test.

The routes are called using the Cypress command - cy.request() just before running assertions. All it does it make an HTTP request to the route and the app state is ready within a fraction of a second.

Earlier, the test suite took over 1:10 minutes and with the latest approach, it finishes between 29-34 seconds.

Let me know your thoughts on this, if we can find more ways to improvise this.

cc: @10up/open-source-practice