10.3.0 regression - test runner randomly fails when running electron locally

[glomotion]

Current behavior

So when running COMPONENT tests specs locally, the test runner randomly crashes out at varying specs. 🤔

ERR_FAILED (-2) loading 'http://localhost:51860/__/#/specs/runner?file=src/components/section-heading/section-heading.test.tsx'
Error: ERR_FAILED (-2) loading 'http://localhost:51860/__/#/specs/runner?file=src/components/section-heading/section-heading.test.tsx'
    at rejectAndCleanup (node:electron/js2c/browser_init:161:7647)
    at EventEmitter.stopLoadingListener (node:electron/js2c/browser_init:161:8022)
    at EventEmitter.emit (node:events:390:28)
    at EventEmitter.emit (node:domain:475:12)

Sometimes it fails at other specs. I have tested every single spec where it has crashed using cypress open - and all specs pass when run manually. I think there's something up with the test runner inside of MACOS 12.4 (21F79)

Sometimes the test runner gets as far as (7 of 64) sometimes (16 of 64), and sometimes it gets to (26 of 64). 🤔 Im having a tough time trying to work out why the runner crashes out at various different specs, there is almost nothing else running while the cypress runner is working.

Desired behaviour

for all the specs to run, just like they do in CI

Test code to reproduce

Ill try and update this ticket with these details, but was hoping to gauge how many people are also finding this issue right now

Cypress Version

10.3.0

Other

Some other relevant dependencies:

"@emotion/css": "^11.5.0",
"react": "^17.0.2",
"react-dom": "^17.0.2"
"cypress-real-events": "^1.7.1",
"@cypress/code-coverage": "^3.10.0",

MacOS 12.4 (21F79) running on an M1 MAX

[glomotion]

whats very strange, is that running the tests with coverage turned on, the test runner gets through all tests, without crashing. When i disable code coverage, that when the test runner randomly crashes out. So weird.

[lmiller1990]

Hi Tim - thanks for the bug report, your issues are always informative.

Normally the issue is the opposite - CI fails, local passes. Disabling code coverage -> success is even more weird. What plugins/libs/versions are you using for code coverage?

Does this happen in the previous version of Cypress? If it really was introduced 10.3.0, that's a small surface area, it should be easy to fix - can you provide a minimal reproduction, along with answers to the above questions?

[glomotion]

Hey again Lachlan :) (and awww thank you!)

OK so to try and answer your questions:

• It now seems I was mistaken about the test runner NOT crashing locally with coverage enabled - it just crashed on me then while coverage was enabled.
• FYI we use "@cypress/code-coverage": "^3.10.0", (i'll add that dep to the OP above).
• This definitely did NOT happen when we were on 9.6.x - the local test runner was bullet proof.

As for the repro, that will take some time (the repo in question is private, sadly 😢). I'll see if i can throw something together in the next day or so and report back.

[glomotion]

So i've been trying to create a stripped down version of the repo in question - to demonstrate the error happening - but it seems that with less specs - i cannot get the issue to happen. :(

And I don't think i can show you a demo where the issue is happening because id have to move a whole private design system repository into a public repo - just to show the error happening.

Whilst trying to provide the repro - i've tried deleting any specs which cause the error - 1 by 1 - but each time it will randomly crash on a new spec. Again, each of the specs that it crashes out on, actually pass when i run them manually.

Something which is worth noting: whenever i run the cypress test runner locally (in mac os) after upgrading to 10.x - i get the following error log at the start:

objc[48773]: Class WebSwapCGLLayer is implemented in both /System/Library/Frameworks/WebKit.framework/Versions/A/Frameworks/WebCore.framework/Versions/A/Frameworks/libANGLE-shared.dylib (0x253f4f808) and /Users/tim/Library/Caches/Cypress/10.3.0/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libGLESv2.dylib (0x109a313c8). One of the two will be used. Which one is undefined.

I overlooked that above error before, because the test runner seems to continue on afterwards - but maybe theres more to this error output than i first clocked.

Ive also tried upgrading all the components to use react 18.x - this also does not help.

Honestly it feels like it's some kind of weird memory related issues, because it doesn't seem to start happening until there are lots of specs (~64 or so).

[glomotion]

hey there again @lmiller1990! So i had an idea overnight, and have actually managed to strip down the repo to just a few components and tests, then duplicated everything 3 or 4 times over ... so that this repo now replicates the test runner issue we're seeing locally on our end 🎉

https://github.com/glomotion/test-cypress-typescript-react/

when you run npm t you should see the test runner crash out at a random point like this:

(its a private repo, so only you and I have axs to it right now)

[lmiller1990]

@glomotion Right, that's what the GH repo invite was. (Was out for 1 week).

I will try running this and see if I run into the same problem. Having a minimal repro makes it sooooo much easier to debug and by extension much more likely the problem is solved, thanks for this!

[lmiller1990]

@glomotion are you able to invite @BlueWinds to your repo, too? They will be taking a look at this one. Thanks!

[glomotion]

@BlueWinds & @lmiller1990 done! :)

[BlueWinds]

Just writing down some things I've tried to help narrow down the problem.

• Crashes completely reliably on Kubuntu 22.04, so not mac / M1 related.
• Removing code coverage did not fix it for me - I still get reliable crashes even with that gone.
• --headed: Still crashes.
• --browser chromium / --browser firefox: No crash. (tests fail on firefox, but that's unrelated)
• Cypress 10.0.0: Still crashes.

So in short, it's electron specific, operating system independent, and not related to the 18.0.4 -> 18.3.0 Electron upgrade that happened in Cy 10.2.0 (which was my guess).

I ran the tests with DEBUG=*, but this didn't yield anything useful - electron just straight up dies when we're asking it to load the next spec, no message beyond the one you already posted.

electron/electron#28208 (comment) is the only potentially relevant Electron bug I was able to find - specifically:

As an additional case, this error seems to happen if the page code has an onbeforeunload event handler that blocks the unload. Then the same 'GUEST_VIEW_MANAGER_CALL': Error: ERR_FAILED (-2) happens and the loadURL fails.

sounds like it could be potentially relevant.

@glomotion - What's the last 'known good' Cypress version (if any) where you weren't running into this?

[BlueWinds]

I should also note that I'm seeing this crash super reliably right after the 32nd spec, rather than jumping around to various locations as you described. Could possibly be an OS difference.

[glomotion]

@BlueWinds it seems all versions that we've tried of 9.x (eg 9.6.1) are totally fine. We only spotted this issue after migrating one of our codebases to cypress 10.x

[lmiller1990]

@BlueWinds that's interesting, glad we can at least reproduce the "crashes after X specs" bug. There is someone else having the same issue (after 138 specs): #22208

They reported better results w/ Chrome. Either way these issues look suspiciously similar.

[BlueWinds]

Yeah, it appears to work fine under Chrome, no crashes at all, so that's a workaround. Added 'electron' to the title to help others locate the issue.

[chenxsan]

Hmm good point, this got me thinking... it's definitely possible to use a different version of Electron if you have it installed locally. Documentation.

We should be able to isolate this. First, we need to answer the question:

• is this an Electron version issue (eg - would downgrading electron even solve the issue)?
• or, is this a Cypress 10 only issue?
• finally, does this only happen with component testing? That is the assumption here - I didn't think about it too much, but if this is a CT only issue, that narrows the surface area a lot.

Downgrading the electron version we ship is a little more involved in just changing a dependency, but maybe we debug it like this:

1. Run Cypress 10 and passing different electron versions, like documented: cypress run --component --browser /usr/bin/some-electron-version
2. Try lots of Electrons
3. Try the same thing, but with Cypress 9
4. Isolate issue!

Good input - I feel like we are slowly closing in on this.

We ran into a similar problem when upgrading cypress 10.7.0 to 10.8.0, where tests are not component testing. Using chrome browser could work around the problem, see webpack/webpack.js.org#6415. But I'm not quite sure that it's electron related since cypress has same electron version in both 10.7.0 and 10.8.0, and we didn't run into any issue with 10.7.0 before.

[lmiller1990]

@chenxsan did you see CI fail multiple times or just once? I wonder if it was just flake - I posted in your PR with the dif from 10.7->10.8 but I don't see any obvious thing that would impact your tests.

Can you reproduce locally or only on CI?

[chenxsan]

@chenxsan did you see CI fail multiple times or just once? I wonder if it was just flake - I posted in your PR with the dif from 10.7->10.8 but I don't see any obvious thing that would impact your tests.

Can you reproduce locally or only on CI?

I've tried that failed CI many times, all of them failed.

Yeah, that's why I was surprised that using chrome wouldn't throw any errors.

I have some network problems here when fetching some public api so I can't run the code locally.

[lmiller1990]

@chenxsan i think thien is related #23830

[mysticdevx]

Hi,
I have the same issue with the following:
Cypress v12.1.0
Node v14.17.3

I have multiple projects in the monorepo, and electron starts with no problem for all projects except one locally. The issue started all of a sudden, with no change in the project. The failing project is working fine in CI and with other browsers locally. I am afraid it will spread like a pandemic :(

Here is the stack trace I got:

Error: ERR_FAILED (-2) loading 'https://my-tesit-site.com/__/'
    at rejectAndCleanup (node:electron/js2c/browser_init:165:7500)
    at EventEmitter.stopLoadingListener (node:electron/js2c/browser_init:165:7875)
    at EventEmitter.emit (node:events:527:28)

i hope there will be a workaround.

[lmiller1990]

I have multiple projects in the monorepo, and electron starts with no problem for all projects except one locally. The issue started all of a sudden, with no change in the project,

Can you share some code? There must be something unique about that one project - some combination of deps, or something. You can try running DEBUG=cypress:* yarn cypress ... to get more verbose logging.

Was this introduced with Cy 12?

[mysticdevx]

@lmiller1990
it occurred after the cypress upgrade from 11.1.0 to 12.1.0. More interestingly, it ran several times successfully and stopped running without any change in the code.
There's no difference with that project (and not possible to have any) bc all the settings are extended from the same directory as other projects, i.e, my structure is:

-packages
|--cypress-config
|--myPrj1
|--myPrj2
|--myPrj3
|--myPrj4
|--myPrj5 // fails opening the electron browser

Where all the cypress-related files reside within packages/cypress-config, and all the projects extend or import the necessary files and configuration.

Below is the related debug log.

cylog.txt

The stack trace I posted above does not appear in logs; it's from the cypress open window when I close the electron.
What I tried to fix: I have reinstalled all the packages in my project, cleared the node and cypress cache and reinstalled all the dependencies along with the cypress. But unfortunately still the same.

[lmiller1990]

it ran several times successfully and stopped running without any change in the code.

It just fails in the middle of a run? Your logs say

  //// all the stats about processor and memory, long bc i have spend some time to wait
  //// then i have manually closed the blank electron screen here

What do you see on the screen when you close Electron? I initially thought it was closing itself (by crashing) but your comment implies you are closing it manually?

[mysticdevx]

What do you see on the screen when you close Electron? I initially thought it was closing itself (by crashing) but your comment implies you are closing it manually?

@lmiller1990 it does not fail in the middle of the run. Once I select the electron browser and hit start, then it does not load the test runner. It hangs on in blank screen for a long time, then I manually close the electron. I'm attaching the two SS. (as stated above, this does not happen in any of the other 4 projects residing in the same repo.)

ss 1: electron hung on during start

ss 2: I have manually closed the electron bc it hung on for about 5 mins.

[lmiller1990]

Can you share the cypress.config.ts for that project? It looks like something is failing in the child process where the config is executed.

[DamienCassou]

I can reliably reproduce this issue with Electron (other browsers work fine) both locally and on CI for an E2E test. Also, Cypress 10.7 works reliably even with Electron. It's only Cypress >= 10.8 + Electron which causes trouble for me.

[lmiller1990]

Are you able to share a repo that reproduces this (perhaps on GH Actions?) @DamienCassou? Knowing Cypress 10.8 introduce the issue is very useful, thanks for narrowing this down.

The diff between 10.7 and 10.8: v10.7.0...v10.8.0 not huge. I don't have the bandwidth to go through this right now, but when I do (or someone else wants to) this is a good place to start.

[mysticdevx]

Can you share the cypress.config.ts for that project? It looks like something is failing in the child process where the config is executed.

Hi, @lmiller1990
I was trying to create a reproducible example, but unfortunately, I could not. The cypress.config.ts and all the cypress-related files (support, index etc.) are the same for all projects.
Then I have solved my problem by deleting the Cypress folder under the Application Support dir. When I track it down, I found that the following directory is the root cause: /Users/<myUsername>/Library/Application Support/Cypress/Partitions/interactive/Service Worker. Unfortunately, I could not track down which file under it caused the issue.

I hope it helps the resolution of the issue in general, or at least for the ones who face similar to my case.

[DamienCassou]

@lmiller1990 I tried simplifying my test and code base but I failed. This particular test is doing a lot of things that seems required to make the test crash. The only thing I managed to understand is that the crash happens when clicking on a link with an href targetting a downloadable file.

[lmiller1990]

@DamienCassou oh interesting, that's really useful info to help debug.

I won't be able to dig into this right now - I'm sorry - but I am happy to help review/give pointers if anyone else is interested in looking into it.

[jennifer-shehane]

Is this still an issue?

[mysticdevx]

Can you share the cypress.config.ts for that project? It looks like something is failing in the child process where the config is executed.

Hi, @lmiller1990 I was trying to create a reproducible example, but unfortunately, I could not. The cypress.config.ts and all the cypress-related files (support, index etc.) are the same for all projects. Then I have solved my problem by deleting the Cypress folder under the Application Support dir. When I track it down, I found that the following directory is the root cause: /Users/<myUsername>/Library/Application Support/Cypress/Partitions/interactive/Service Worker. Unfortunately, I could not track down which file under it caused the issue.

I hope it helps the resolution of the issue in general, or at least for the ones who face similar to my case.

@jennifer-shehane for me, after this, the issue didn’t appeared. (Currently I am in the latest version)

[PrakratiAgarwal123]

@jennifer-shehane Solution provided does not work for me since I dont have the path mentioned in my system-/Users//Library/Application Support/Cypress/Partitions/interactive/Service Worker