E2E tests are running extremely slow

[agentdon]

Current behavior

Our E2E tests are running extremely slow right now. We have a Vue app that we just switched over to Vite from vue-cli. The slow down seems to be related to using Vite as that's immediately when we noticed it. However, loading the app locally outside of Cypress is extremely fast, so not sure if Vite is actually the issue here.

Edit

As @rubber-duck-software mentioned below, the initial load is super slow - also > 90 seconds for me. And Cypress does appear to be requesting files one at a time. There seems to be a little bit of an improvement in the test run time after the first run. However, this improvement seems really spotty as the page load still times out most of the time.

When running vite dev it appears that Cypress is requesting files one at a time. The first test run, this is remarkably slow ( > 90 seconds). On subsequent test runs this is more like 45 seconds as vite responds primarily with 304s.

Desired behavior

Tests run quickly like they used to before switching to Vite.

Test code to reproduce

Unfortunately, not sure how to provide guidance on how to reproduce. I tried to setup an example repo to reproduce, but this issue seems to be specific to our app. Happy to provide additional details that are helpful. I've attached a zip file that contains our Cypress files, config, package.json, and yarn.lock. Please let me know if there are any additional details that I can provide that are useful for debugging.

cypress-bug.zip

Cypress Version

10.3.1

Other

Vue version: 2.6.14
Vite version: 3.0.2

[rubber-duck-software]

Running into a similar issue with a react project which builds using Vite. When running vite dev it appears that Cypress is requesting files one at a time. The first test run, this is remarkably slow ( > 90 seconds). On subsequent test runs this is more like 45 seconds as vite responds primarily with 304s. Current workaround is running Vite build then Vite preview which appears to resolve the issue, but is a very slow development cycle.

[mverdaguer]

We are facing a similar problem with component testing.

In our case we have a monorepo with a structure like:

• custom-ui-framework.
• app-a (depends on custom-ui-framework).
• app-b (depends on custom-ui-framework).

We are using:

• cypress 10.3.1 + @cypress/vue2 1.0.1
• vue 2.6.14.
• vite 2.9.14.

We can notice that when running component tests on any package, all the components from the custom-ui-framework are being fetched, no matter if they are being used or not by the tests.

[agentdon]

@rubber-duck-software Glad you mentioned the files loading one at a time and the extremely long initial start time. I'm seeing the same thing as well. However, in our case the page loads still hit the 60 second timeout most of the time. I updated the issue description to capture this information.

[agentdon]

@mverdaguer There's also an open issue for slow component tests here. Not sure if you saw that or not, but it might be helpful to post the information in that issue and follow along there. From what the Cypress team is saying, it looks like these issues are related, but different.

[lmiller1990]

I think I misunderstood this issue when it was mentioned (iirc in another issue, not sure if by a different user). To clarify you are writing E2E tests against an app running in dev mode with yarn vite? So - the bottleneck is whatever is happening between the fetch requests, like the Cypress network layer (used for things like cy.intercept, etc?

If you build the app and run the tests against the production build (not the version of the app running with yarn vite dev for example, is it nice and quick? Or are you experiencing the slowness, even against the production build?

[agentdon]

Correct.

To clarify you are writing E2E tests against an app running in dev mode with yarn vite?

I'm not exactly sure what the bottleneck is. Like @rubber-duck-software mentioned, I noticed that the files seem to be requested one at a time through Cypress. However, my understanding of Vite is that this is how things are supposed to function since it delegates most module request to the browser's native ES module functionality.

So - the bottleneck is whatever is happening between the fetch requests, like the Cypress network layer (used for things like cy.intercept, etc?

The tests are nice and quick when using vite preview - which uses a static JS bundle. It's just when using vite dev and running the tests through Cypress that things slow to a crawl and timeout. Notably, the app loads quickly when serving it with vite dev and accessing it normally through the browser.

If you build the app and run the tests against the production build (not the version of the app running with yarn vite dev for example, is it nice and quick? Or are you experiencing the slowness, even against the production build?

From what I'm seeing, there appears to be some interaction between Cypress and Vite causing the slow down.

[lmiller1990]

Okay, I think I understand what you are describing now. Just to clarify:

If you build the app and run the tests against the production build (not the version of the app running with yarn vite dev for example, is it nice and quick? Or are you experiencing the slowness, even against the production build?

You responded:

From what I'm seeing, there appears to be some interaction between Cypress and Vite causing the slow down.

There is one more thing you can do to narrow this down, which will help a lot. Right now, I think you are running your E2E tests against the Vite development server. I'm sure Vite does different things in dev mode that it does not do when you build for production.

If you build your app, say yarn vite build and then serve the static assets via a non dev server (like NGINX, or npx serve, or whatever - some standard HTTP server, basically what you do in production) and run your E2E suite against the production build, is it still nice and fast? An alternative would be just pointing Cypress at your staging (or production, if you are brave) deployment and see if the issue continues. I

Basically, I'm trying to narrow it down - is it a general Vite issue, or a Vite-but-only-when-used-in-development-mode issue?

Current theory is either Vite does different things in development mode, like lots of pre-fetching or optimization, causing some kind of slowdown when used in Cypress, or there's some bottleneck when serving Vite's assets through Cypress (Cypress has an entire network layer that all requests pass through, to facilitate things like cy.intercept. This includes web sockets, which Vite uses during development mode for hot reload, etc.

[agentdon]

Sorry, I wasn't clear in my previous message. 🙂 I'm 99% sure this is a Vite-but-only-when-used-in-development-mode issue. I only encounter this problem when running yarn vite dev. As you suggested, I already tried doing yarn vite build and then yarn vite preview (which basically just serves the prebuilt app through an HTTP server) and things run quickly. So, this appears to be a Vite dev serve / Cypress interaction.

There is one more thing you can do to narrow this down, which will help a lot. Right now, I think you are running your E2E tests against the Vite development server. I'm sure Vite does different things in dev mode that it does not do when you build for production.

If you build your app, say yarn vite build and then serve the static assets via a non dev server (like NGINX, or npx serve, or whatever - some standard HTTP server, basically what you do in production) and run your E2E suite against the production build, is it still nice and fast? An alternative would be just pointing Cypress at your staging (or production, if you are brave) deployment and see if the issue continues. I

Basically, I'm trying to narrow it down - is it a general Vite issue, or a Vite-but-only-when-used-in-development-mode issue?

I agree. Based on @rubber-duck-software's observation that Vite appears to be doing a lot of single file fetching, it seems fairly likely that things are slowing down at the network layer in Cypress.

Current theory is either Vite does different things in development mode, like lots of pre-fetching or optimization, causing some kind of slowdown when used in Cypress, or there's some bottleneck when serving Vite's assets through Cypress (Cypress has an entire network layer that all requests pass through, to facilitate things like cy.intercept. This includes web sockets, which Vite uses during development mode for hot reload, etc.

Please let me know if there's anything else I can do to help debug this. This issue is preventing us from using Cypress right now. Well, on that note, is there a workaround in the meantime while we're waiting on a fix?

[lmiller1990]

I'm 99% sure this is a Vite-but-only-when-used-in-development-mode issue

This is useful, and I think based on what you've suggested, you are correct.

Please let me know if there's anything else I can do to help debug this.

I think you've done great with the info you've provided. I know about as much about the Cypress network layer as you do, so I guess we just need to do a source code dive.

The next step I'd take with debugging is to run with DEBUG=cypress:* yarn cypress... and watch the (huge amount of) logs, and see if there's any obvious bottleneck (eg, they stop printing, or print really slow, at some specific point).

Also, I'd like to learn the difference between Vite dev vs Vite prod mode. The fact this happens in Vite dev mode but not production tells me we should find out what's different.

This issue is preventing us from using Cypress right now.

:( sorry to hear, hope we can resolve this soon.

Well, on that note, is there a workaround in the meantime while we're waiting on a fix?

I don't know of a good work around at this point, sorry!

Going to tag @flotwig since he might know something off the top of his head that would explain this interaction between Vite (which just serves ES modules via HTTP, apparently many tiny requests) and Cypress' network layer.

[agentdon]

Before I respond to the rest of your comments, when I opened up Cypress to run things in debug mode as you suggested, I noticed that things seem to be loading faster. At least to the point now where page loads are only taking 15-20 seconds as opposed to hitting the 60 second timeout that I mentioned previously. I dug through our new commits and unfortunately couldn't figure out what caused this speed up. While things are faster, the workflow is still drastically slower than what we had before the Vite upgrade. Now, on to responding to your comments.

I ran in debug mode. I captured the output in this file(which I also added to the original post above) and also recorded this video. I do see a couple of slow downs. However, I think it's worth noting that those slow downs seem to be after the page has loaded (i. e. they don't seem to be related).

The next step I'd take with debugging is to run with DEBUG=cypress:* yarn cypress... and watch the (huge amount of) logs, and see if there's any obvious bottleneck (eg, they stop printing, or print really slow, at some specific point).

I came across this StackOverflow post which matches my understanding of the difference between the two. I think it's also beneficial to add that Vite dev mode means the browser will be doing much of the work since Vite delegates module loading to the browser's native ES module loading functionality.

Also, I'd like to learn the difference between Vite dev vs Vite prod mode. The fact this happens in Vite dev mode but not production tells me we should find out what's different.

No worries! As I reread my previous comment, I realized it may have come across salty. 😬 Sorry about that. I really appreciate your help trying to resolve this and didn't mean to come across negatively at all. I was just trying to inform you that this issue was making Cypress totally unusable for us (prior to the speed up I mentioned at the start of this comment). 😄 With the minor speed up, it now seems to be somewhat usable, but has still slowed down our workflow dramatically compared to what we had before.

:( sorry to hear, hope we can resolve this soon.

Great, thanks! @flotwig if you know of any workarounds that we can use while we work through this issue, that would be much appreciated. 🙏

Going to tag @flotwig since he might know something off the top of his head that would explain this interaction between Vite (which just serves ES modules via HTTP, apparently many tiny requests) and Cypress' network layer.

[Threnos]

Bump, I have also experienced this. Had to switch to serving via "vite preview" until this is sorted out.

[lmiller1990]

Vite is a build tool that enables faster development by re-compiling only the changed files on each save, and using a simple development server that supports hot module replacement (HMR).
Vite preview is a CLI utility that can be used to preview Vite projects in a production-like environment. It builds the project, starts a production server, and opens a browser to the server URL.

So vite preview is working properly? This does confirm that it's a Vite dev mode only problem, which is pretty interesting.

[lmiller1990]

I think these two might be related: #22868

Projects that serve lots of assets (100s of network requests) take a long time to load. It's not obvious normally when you visit your dev server, since your browser is caching most of those, but Cypress clears the cache before each spec, so all the requests fire.

[lmiller1990]

Maybe fixed by #25209. Could be dup of #22868.

[Narretz]

There are a few things that would improve this:

• Experimental feature: disableProxy #25201 - disable the proxy for specific paths which reduces the proxy overhead
• Add HTTP/2 Support in Proxy Layer #3708 - would allow parallel download of module files
• Allow for the browser caching of files during tests #18335 - vite will invalidate hot module reloaded files by appending a timestamp query param. And during testing of multiple specs, the code doesn't change anyway, so cypress could cache all module files once it has encountered them

I'm not sure if you can do anything from the vitejs site, serving the modules directly kinda is the main selling point of vite dev server

[lmiller1990]

Lack of HTTP/2 is most likely 90% of the problem, need multiplexing for the huge amount of JS modules, once that is supported in Cypress (that issue is marked "in progress", someone working on it internally) things will be nice and fast and good.

[agentdon]

Lack of HTTP/2 is most likely 90% of the problem, need multiplexing for the huge amount of JS modules, once that is supported in Cypress (that issue is marked "in progress", someone working on it internally) things will be nice and fast and good.

@lmiller1990 👋 Do you have a link to HTTP/2 issue? I upgraded to the most recent version of Cypress and the tests appear to be running normally, so I'm wondering if the fix landed?

[lmiller1990]

HTTP/2 issue: #3708

The person who was working on this is no longer with the organization, I'm not sure if any additional progress has been made since, or if someone else picked it up.

[agentdon]

Thanks for the quick response Lachlan! Ah, gotcha. Is there someone I can reach out to follow-up on the issue to see where it's at? I thought the upgrade fixed our slow Cypress tests, but it seems that the upgrade just fixed something temporarily and now they've slowed down again. Because the tests run so slow, this is issue is preventing us from using Cypress for anything other than a few basic E2E tests for functionality that we can't cover with Vitest. I'd like to find out where everything stands to see if we might be able to start seriously using Cypress. 😄

For what it's worth, I wonder if cache or something along those lines is the culprit or contributing to this issue. The issue seemed to correct itself for a bit after I upgraded. Does that spark any ideas on things that I could try to see if cache or something along those lines is the problem?

[lmiller1990]

The issue seemed to correct itself for a bit after I upgraded

This is strange. I wouldn't expect this to "fix" itself. It's usually either fast, or it's not. I'm not too sure what would explain this.

I don't think there's any low hanging fruit to optimize for a large number of modules, except perhaps testing against a built app (eg yarn vite build as opposed to a development mode one using a dev server.

[agentdon]

We have tried using vite build to run the tests and that does fix the performance problems. However, waiting for the app to rebuild before every test provided to be super-time consuming. I think it comes out to about the same amount of time as running Cypress against the Vite dev server. 😞 Do you know if there's a plan to do any more work on this? Is there someone that I can check-in with to see where everything is at?

[Narretz]

HTTP/2 issue: #3708

The person who was working on this is no longer with the organization, I'm not sure if any additional progress has been made since, or if someone else picked it up.

This is very unfortunate. I was under the impression http2 support was on the roadmap and actively being worked on, since it seems like it could benefit quite a few use cases. Is there any way to put this back in progress or let the cypress team know that this is really valuable?

[lmiller1990]

I took a look at the App Priority Board which should list all the in-flight big ticket items, and HTTP/2 isn't there right now. I can verify with our product team to sure this is up to date, I'm sure it is -- but always good to double check. I agree HTTP/2 would be very beneficial - I think everyone is on the same page.

As for raising awareness, I think using the original issue and 👍 is really the best avenue. I understand this doesn't feel great (I've done the same thing in other projects for issues I really want to see solved).

[agentdon]

Ah, okay got it. I just gave it a thumbs up. I'm bummed to hear this dropped off the roadmap as I think this is unfortunately going to prevent us from using Cypress seriously for anything other than a couple of smoke tests, but I understand how the prioritization thing goes. Thanks for your help Lachlan! 🙏

[amcsi]

So Vite - the number 1 go-to bundler for React and Vue development - is not properly handled by Cypress, and results in tests simply running super slow? I'm surprised people aren't making a bigger deal out of this, assuming Cypress is a popular testing framework with a vocal community.

[lmiller1990]

I think the performance implications greatly depend on your app, how many modules, if you are testing against a dev or prod build... certainly this is a big issue for some people.

Also worth remembering that while Vite is popular for new apps, many apps are either:

1. webpack
2. server rendered (rails, php, django, java/spring...)

I suspect Vite is not as prevalent as we JS / frontend devs like to think. I sure hope it does become more popular, it's fast!!

Either way, it should would be good to see if there's anything we can do short term / low hanging fruit, since while HTTP/2 updates would be really great, it's proven to be a non trivial change to a complex and widely used product, which is why it's taking a long time to implement.

[AriPerkkio]

I've been running into the same exact issue where Cypress is really slow against vite dev server. However it's possible to get decent developer experience by using vite build with --watch flag. On my setup the initial build takes a while but builds triggered by file changes are happening fast. Not as fast as with vite dev but it's fine.

This can all be integrated into Cypress configuration so that only cypress command has to be run. Example below runs vite build and vite preview during Cypress start up and closes the server before Cypress shuts down. When using cypress open for test development it sets up vite build with --watch flag so that changes in codebase are triggering re-builds.

import {defineConfig} from 'cypress';
import {build, preview} from 'vite';

// This is "cypress open" when developing tests and "cypress run" when just running tests, e.g. CI
const IS_INTERACTIVE = process.env.npm_lifecycle_script?.includes('cypress open');

export default defineConfig({
  e2e: {
    baseUrl: 'http://localhost:3000',
    async setupNodeEvents(on) {
      console.log(`Starting Vite server${IS_INTERACTIVE ? ' in watch mode' : ''}.`);
      const watcher = await build({build: {watch: IS_INTERACTIVE ? {} : null}});
      const server = await preview({ preview: { port: 3000, strictPort: true } });

      on('after:run', async () => {
        if ('close' in watcher) {
          await watcher.close();
        }

        await new Promise((resolve, reject) => {
          server.httpServer.close(error =>
            error ? reject(error) : resolve())
        })
      });
    },
  },
});

[hackel]

The workaround for slow startup times seems to be to run against a prod build, but is there any way to do this for component tests? Since they integrate with Vite directly, I don't see a way to do that. In my case, it takes over 30 seconds for the modules to load, and then the test runs quickly in under 1 second.

[santoshD93]

facing the same problem.

[pgross41]

I cobbled together a solution workaround that has our app loading with 2,000+ requests in Cypress Chrome equally as fast as regular Chrome. This is an implementation of the idea posted here: #3708

Short version:

1. Configure a special host name in your hosts file e.g. localhost-no-cypress-proxy
2. Configure Cypress to run Chrome with --localhost-no-cypress-proxy to include the special hostname
3. Configure Vite to modify index.html to use the special host for source code files

Long version:

Configure a special host name in your hosts file e.g. localhost-no-cypress-proxy

Edit the device's hosts file so that it includes the following to map the special hostname to localhost:

127.0.0.1 localhost-no-cypress-proxy

Windows location: C:\Windows\System32\drivers\etc\hosts
macOS and Linux: /etc/hosts

Configure Cypress to run Chrome with --localhost-no-cypress-proxy to include the special hostname

Add the following to Cypress config:

{
  // ...
  e2e: {
    setupNodeEvents(on, config) {
      on('before:browser:launch', (browser = {} as Browser, launchOptions) => {
        if (browser.name === 'chrome') {
            // Anything hitting `http://localhost-no-cypress-proxy` will bypess the Cypress proxy
            const indexOf = launchOptions.args.indexOf('--proxy-bypass-list=<-loopback>');
            const proxyByPassList = ['<-loopback>', 'localhost-no-cypress-proxy'];
            const proxyByPassListFlag = `--proxy-bypass-list=${proxyByPassList.join(';')}`;
            launchOptions.args[indexOf] = proxyByPassListFlag;
        }
        return launchOptions;
      });
    // ...
    },
  }
// ...
}

Configure Vite to modify index.html to use the special host for source code files

Add the following plugin to the plugins section of Vite config:

import { defineConfig, loadEnv, PluginOption } from 'vite';

export default defineConfig(({ mode }) => {
    return {
        plugins: [ enableLocalhostNoCypressProxyPlugin(mode) ],
        // ...
    }
});

function enableLocalhostNoCypressProxyPlugin(mode: string): PluginOption {
    return {
        name: 'enable-localhost-no-cypress-proxy-plugin',
        transformIndexHtml: {
            enforce: 'pre', // Run this before other processes
            transform() {
                const viteEnv = loadEnv(mode, process.cwd());
                if (viteEnv.VITE_INDEX_HTML_CYPRESS_NO_PROXY) {
                    // Modify index.html to use the special host that cypress chrome doesn't proxy
                    const indexHtml = fs.readFileSync('./index.html', { encoding: 'utf8' });
                    return indexHtml.replace(
                        '/src/index.tsx',
                        'http://localhost-no-cypress-proxy:3000/src/index.tsx'
                    );
                }
            },
        },
    };
}

The if statement (viteEnv.VITE_INDEX_HTML_CYPRESS_NO_PROXY) makes it easy to opt in or out of this behavior. To set to true, add the following to a root file called .env.development.local

VITE_INDEX_HTML_CYPRESS_NO_PROXY=true

[amcsi]

I tried to do what @pgross41 wrote, but I'm getting Uncaught Error: React refresh runtime was loaded twice. Maybe you forgot the base path?, and have no idea how to fix it.

I'm using React.

[o-mega]

Is there any updates? Using vite dev server with cypress is not possible at the moment, because of vite pre compile policy that loads all files and includes on first start.

[profispojka]

This would definitely help a lot. I would need to go back to vue-cli in our project, which is in maintenance mode.

[RoniNiklas]

For context, our React app had a 75% reduction in Cypress test runtime by changing from running Cypress tests against the vite-dev-server to running against vite build && vite serve.

If you still prefer running against the vite-dev-server, you can override the cache-control headers to make the browser cache the app's files. This gave a 25% reduction in test runtime. The first test of a test file runs as slowly as before, but the rest run significantly faster, as the app is cached in the browser. Cypress discards the cache between test files.

Add this plugin to your vite.config.ts > config > defineConfig > plugins

// Need a separate plugin since vite-dev-server's headers option does not apply the headers to all files reliably
function cacheControlPlugin(): Plugin {
  return {
    name: 'vite-plugin-cache-control',
    apply: 'serve',
    enforce: 'post',

    configureServer(server) {
      server.middlewares.use((req, res, next) => {
        // Cache all files in browser for 30 seconds.
        // Vite handles cache-busting by appending ?t="timestamp" in the url when you modify files
        // so this should not cause issues except when restarting the server (when the timestamp is reset)
        res.setHeader('cache-control', 'max-age=30, immutable')
        next()
      })
    },
  }
}

Note that while vite does timestamp changes to component files, which invalidates the cache whenever you modify a component, it does not timestamp files when the dev-server restarts, so the cache needs to be short enough that you don't run into issues with the dev-server serving old files. And your developers need to be aware of the cache.