Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing browserCriClient in connectToNewSpec #24650

Closed
jazpearson opened this issue Nov 10, 2022 · 39 comments · Fixed by #27116
Closed

Missing browserCriClient in connectToNewSpec #24650

jazpearson opened this issue Nov 10, 2022 · 39 comments · Fixed by #27116
Labels
E2E Issue related to end-to-end testing Triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. type: regression A bug that didn't appear until a specific Cy version release

Comments

@jazpearson
Copy link

Current behavior

Unfortunately, this issue is intermittent and only happens in our CI pipeline. But i wanted to see if could raise the issue in case there's something obvious.

We run cypress with the following: cypress-run cross-env NODE_OPTIONS=--max-http-header-size=1000000 cypress run --browser chrome. And we run in a docker container, using the cypress/included images.

This error happens mid-way running through our test suites. Our tests can take 30 minutes, and the following error seems to occur when ending a test suite (or conversely, when starting the next test suite).

Timed out waiting for the browser to connect. Retrying...
Missing browserCriClient in connectToNewSpec
Error: Missing browserCriClient in connectToNewSpec
    at Object.connectToNewSpec (<embedded>:2744:242702)
    at Object.connectToNewSpec (<embedded>:2750:24813)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async v.relaunchBrowser (<embedded>:2802:346537)

When this happens, the test run completely exits and the remaining suites do not run.

Desired behavior

For the test run not to crash

Test code to reproduce

Really difficult to do, sorry. It's intermittent, and it's a fairly large, sensitive, code-base.

Cypress Version

11.0.1

Node version

16.16.0

Operating System

Jenkins CI - CentOS (but runs the cypress docker image)

Debug Logs

No response

Other

We've seen this happening in v10 and v11.

@rsouthgate
Copy link

@jazpearson Don't know if this helps but stumbled across this when trying to figure out why our own CI pipeline was breaking with this error message after upgrading agents to 10.11.0. In our case the error was being caused by our test director (sorry-cypress) and ultimately was due to this off-by-one error sorry-cypress/sorry-cypress#613 which has since been fixed.

@nagash77
Copy link
Contributor

Hi @jazpearson , I'm sorry to hear you are having problems using Cypress. Unfortunately without a reproducible example, we wont be able to further assist you. If you can figure out a way to reproduce this issue that you can share with us please do comment here and we can reopen the issue.

@nagash77 nagash77 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 14, 2022
@tommy0831
Copy link

I am also seeing this error with Cypress 10.11 when running on Jenkins pipeline. This happens very rarely.

3:11:26 Timed out waiting for the browser to connect. Retrying... 13:11:26 Missing browserCriClient in connectToNewSpec 13:11:26 Error: Missing browserCriClient in connectToNewSpec 13:11:26 at Object.connectToNewSpec (/root/.cache/Cypress/10.11.0/Cypress/resources/app/packages/server/lib/browsers/chrome.js:451:19) 13:11:26 at Object.connectToNewSpec (/root/.cache/Cypress/10.11.0/Cypress/resources/app/packages/server/lib/browsers/index.js:96:31) 13:11:26 at processTicksAndRejections (node:internal/process/task_queues:96:5) 13:11:26 at OpenProject.relaunchBrowser (/root/.cache/Cypress/10.11.0/Cypress/resources/app/packages/server/lib/open_project.js:150:24)

@nagash77
Copy link
Contributor

Hi @tommy0831 , can you provide a reproducible example that our engineers can use to debug further?

@jazpearson
Copy link
Author

One thing i did do was to update these two libraries that i had installed:

  • @cypress/webpack-preprocessor from 4.0.3 to 5.15.7
  • cypress-terminal-report from 4.1.2 to 5.0.0.

I don't know if it was coincidence or not, but that did help improve things. It still happens, but it's reasonably rare at the minute. Maybe happens 1/20 times, whereas before it was sometimes as bad as 1/4.

I wish i could provide a sample, but i'm under NDA.

This is currently some of our reporting 3rd party components.

    "cypress-mochawesome-reporter": "3.2.3",
    "cypress-multi-reporters": "^1.6.2",
    "cypress-terminal-report": "^5.0.0"
    "mocha-junit-reporter": "^2.2.0"

@matanbanin
Copy link

Happened to me also. Anything new on this issue?

@nagash77
Copy link
Contributor

Hi @matanbanin , We still do not have a reproducible example for our Cypress devs to use to debug this issue. If you can provide one I will reopen the ticket for further investigation.

@swatinigam
Copy link

Hello, I am also seeing this error. I am running my tests locally (not in ci). These are the dev dependencies:
"cypress": "10.8.0",
"cypress-cucumber-preprocessor": "^4.2.0",
"cypress-real-events": "^1.7.1",
"cypress-repeat": "^2.3.3",
"cypress-xpath": "1.6.2"
Below is the error:

Timed out waiting for the browser to connect. Retrying...
Missing browserCriClient in connectToNewSpec
Error: Missing browserCriClient in connectToNewSpec
    at Object.connectToNewSpec (/Users/username/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/browsers/chrome.js:425:19)
    at Object.connectToNewSpec (/Users/username/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/browsers/index.js:95:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at OpenProject.relaunchBrowser (/Users/username/Library/Caches/Cypress/10.8.0/Cypress.app/Contents/Resources/app/packages/server/lib/open_project.js:148:24)

I have never seen this error before. All I did was run tests, but today I managed to see it a few times already.

@nagash77
Copy link
Contributor

Hi @swatinigam , Do you have a reproducible example by chance? So far no one has been able to provide us with a reproducible example to help us debug this issue. Without being able to reproduce the issue it is extremely unlikely we will find a solution. If you can share one that would allow us to move this ticket forward.

@matanbanin
Copy link

It's happening just not reproducible. happens for about 2% of builds

@emilyrohrbough
Copy link
Member

This seems to be the same issue as #22451, however the error is being thrown correctly here, where as it's getting one line of code further in #22451... we had identified it as a mistake on how we are re-launching a browser after the first spec if the browser process is corrupt.

This error is occurs here: https://github.com/cypress-io/cypress/blob/7721e2be9d7cd6695a16385f3a8965625da162c4/packages/server/lib/browsers/chrome.ts

@nagash77 nagash77 reopened this Jan 5, 2023
@nagash77 nagash77 added routed-to-e2e type: regression A bug that didn't appear until a specific Cy version release labels Jan 5, 2023
@matanbanin
Copy link

Awesome! any idea on how much time will take to deploy a fix? Is there any safe version to rollback to?

@nagash77 nagash77 added the type: performance 🏃‍♀️ Performance related label Jan 11, 2023
@samixchoumi
Copy link

samixchoumi commented Feb 14, 2023

Hello, i've got the same error for no reason today :
My devDependencies from package.json looks like that :

"devDependencies": {
    "@badeball/cypress-cucumber-preprocessor": "^15.1.3",
    "@bahmutov/cypress-esbuild-preprocessor": "^2.1.5",
    "@shelex/cypress-allure-plugin": "^2.35.2",
    "@typescript-eslint/eslint-plugin": "^5.51.0",
    "@typescript-eslint/parser": "^5.51.0",
    "allure-js-commons": "^2.0.0-beta.26",
    "cypress": "^12.5.1",
    "cypress-multi-reporters": "^1.6.2",
    "cypress-real-events": "^1.7.6",
    "cypress-recurse": "^1.27.0",
    "cypress-slack-reporter": "^1.5.2",
    "cypress-slow-down": "^1.2.1",
    "cypress-time-marks": "^1.4.0",
    "esbuild": "^0.16.17",
    "eslint": "^8.34.0",
    "eslint-plugin-cypress": "^2.12.1",
    "fs-extra": "^11.1.0",
    "mocha": "^10.2.0",
    "mochawesome": "^7.1.3",
    "mochawesome-merge": "^4.2.2",
    "mochawesome-report-generator": "^6.2.0",
    "moment": "^2.29.4",
    "typescript": "^4.9.5"
  },

Any news about that ?
I use this new config into my cypress.config.ts file :
experimentalMemoryManagement: true,

@mkruisman2
Copy link

Any update on this issue? Since a few months we are experiencing the same and currently are getting this error multiple times per day, even with the test suites just running once per hour in CircleCI. We're on Cypress 12.3.0

We're also experiencing issue #747, in case it's related.

@chrisbreiding chrisbreiding self-assigned this Feb 16, 2023
@chrisbreiding
Copy link
Contributor

It's hard to say without a reliable way to reproduce this, but there's a chance this is fixed by #25898. It will be out with the next release, but if you'd like to check it out ahead of time, I'd recommend installing the prerelease build for the latest commit on the develop branch.

@samixchoumi
Copy link

samixchoumi commented Feb 28, 2023

@chrisbreiding hello, i still got the error only on gitlab (of course) after updating Cypress to 12.7.0, and i've no idea on how to reproduce it because, it works sometime, and some other time it fail.
I think it's linked to GitLab parallelization but it's just an idea.

@jazpearson
Copy link
Author

Unfortunately we are still seeing this issue in 12.7.0

@hannahhaken
Copy link

hannahhaken commented Mar 27, 2023

We're still seeing this issue sporadically in our GoCD pipeline and we've upgraded to 12.8.1. No parallelisation being done at our end as well.

@emilyrohrbough emilyrohrbough removed the type: performance 🏃‍♀️ Performance related label Mar 27, 2023
@jazpearson
Copy link
Author

jazpearson commented Mar 29, 2023

Is there anything we can add to help debug this? I know we want a reproducible repo, but it seems most of us aren't in a position to share our code.

This is starting to cripple us, as we've added more tests, the chances of this happening has gone up, and we are having to run tests multiple times to check for passes. Our tests take over an hour to run now.

Seeing this issue in 12.9.0

@narancshab
Copy link

It happens with "cypress": "^12.4.1", too:

other dependencies:

    "@badeball/cypress-cucumber-preprocessor": "15.1.3",
    "@bahmutov/cypress-esbuild-preprocessor": "2.1.3",
    "cucumber-html-reporter": "5.5.0",
    "cypress-file-upload": "5.0.8",
    "cypress-iframe": "^1.0.1",

I think it is somehow related to the cucumber-preprocessor.

I executed locally, in headless mode.

@RuudMeeuwsen
Copy link

I've also seen the issue in several 12.x versions (I'm currently on 12.9.0, where it also occurs).
I'm not using cypress-cucumber-preprocessor, and @jazpearson didn't mention it either, so I guess we can safely rule it out as a potential cause of the issue.
Unfortunately I'm also under NDA, so I can't share my codebase either :(.

@hannahhaken
Copy link

@nagash77 @chrisbreiding we're still seeing this issue. I'm not under any kind of NDA with my company's code and am happy to work with someone from Cypress to show you this issue and let you see our overall setup, if that helps at all?

@Stubennett1
Copy link

@chrisbreiding Any update on this issue? Unfortunately this is rapidly becoming a showstopper for us.

Can anyone provide any suggestions on how to mitigate the issue as we are seeing it almost 100% of our runs in the CI/CD environment.

@hannahhaken
Copy link

hannahhaken commented Apr 12, 2023

We're still seeing this issue 1 in every 10 builds, and have resorted to splitting off a small subset of our tests to run in just the prod stage of our GoCD pipeline instead of running them all just in preprod. We're yet to see the issue occur since we've done this. We have a quick enough feedback loop and release/rollback process so we can react to any issues flagged by our Cypress tests in prod quickly enough.

Not an ideal solution, and won't be an option for all experiencing this issue. However this does highlight that the length of the test suite has something to do with why we were experiencing the issue.

@chrisbreiding my offer still stands to work with you and show you our codebase, pipeline and test setup if any of this info might help you understand what's causing the issue.

@nagash77 nagash77 added E2E Issue related to end-to-end testing Triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. and removed routed-to-e2e labels Apr 19, 2023
@matanbanin
Copy link

matanbanin commented Apr 23, 2023

Still happening in 12.10.0

@hannahhaken
Copy link

hannahhaken commented Apr 28, 2023

Hi everyone, thought I'd give an update on something we've changed which has so far stopped this issue from occurring. We've increased the mem_limit and mem_reservation in our docker image.

Previously it was set to:

mem_limit: 3500m
mem_reservation: 3250m

and now we've updated it to be:

mem_limit: 6000m
mem_reservation: 4000m

Like I said, since this change we haven't seen the browserCriClient issue once in our CI pipeline (we'd never encountered this locally).

Our original memory must have been set a while ago, and we haven't thought to change it until now. We thought it might need bumping up after spotting this in the "Hardware" section of getting started, the part about running tests reliably.
(https://docs.cypress.io/guides/getting-started/installing-cypress#Hardware)

Hopefully others experiencing this issue can give this a try and share if it works for them too. Good luck everyone.

@samixchoumi
Copy link

Any update ? We don't use docker image and it still appearing sometime.

@chrisbreiding chrisbreiding removed their assignment May 15, 2023
@jazpearson
Copy link
Author

jazpearson commented May 15, 2023

We are running 12.11.0 and having to run builds mutliple times to get past this issue. It's becoming unworkable and we've invested so much time into creating these tests, We aren't using docker compose.

@jazpearson
Copy link
Author

#25844 looks like another case was described here, but with reproducible code? but this issue was told it was a duplicate of this one....

@jazpearson
Copy link
Author

I've submitted a PR in the hope to get some attention to this issue, because it's really hurting us now.

@przedab
Copy link

przedab commented Jun 9, 2023

The same happens on version 12.13.0.

@AtofStryker
Copy link
Contributor

Hey @jazpearson . I put together #27116 which I think might fix this issue. I will work on getting a binary built and mentioning it on this issue. In the meantime, I am going to close #26882 since I don't believe it is going to fix your issue as recreating the CRI client in that context will just give the wrong port and refuse the connection. I'll keep this issue up to date once I get a binary!

@jazpearson
Copy link
Author

That's fantasic, thanks @AtofStryker. I wasn't convinced my PR would work, and was more to try and get the ball rolling with someone who knew what they were doing.

Thanks again - and will look forward to testing things out.

@AtofStryker
Copy link
Contributor

AtofStryker commented Jun 26, 2023

I meant to update this on Friday, but the binaries with the fix proposed in #27116 are here. @jazpearson confirmed it looks to be working in #24650 (comment), but would love others to try and confirm!

@matanbanin
Copy link

YAY! Thanks!

@jazpearson
Copy link
Author

In my testing, i did see the issue happening again, but the frequency seemed to be much reduced from before.

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Jul 6, 2023

Released in 12.17.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v12.17.0, please open a new issue.

@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Jul 6, 2023
@AtofStryker
Copy link
Contributor

In my testing, i did see the issue happening again, but the frequency seemed to be much reduced from before.

good to know. There might be an edge case we are missing, but it's going to be hard to determine. If you are able to reproduce it, let us know and we can see how we can improve this fix!

@jennifer-shehane
Copy link
Member

jennifer-shehane commented Jul 1, 2024

There is another fix to a similar error like this that will be coming out in 13.13.0, so please update to that version when it releases. PR: #29663

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
E2E Issue related to end-to-end testing Triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. type: regression A bug that didn't appear until a specific Cy version release
Projects
None yet