Recover from renderer / browser crashes

[brian-mann]

Related to #348.

It is actually possible for Cypress to implement strategies when the renderer (or browser process) crashes during a test run - something like recoverFromRendererCrashes: true by default.

There is already a mechanism for Cypress to "reload" mid-run, rebuild the state of every previous run test, skip over previously run tests, and continue with the next one in line.

In fact this is exactly what cy.visit already does under the hood.

We can utilize this same process upon a renderer / browser process crashing to continue on with the run.

So it may look something like this:

(Running Tests)

✓ test 1 - foo
✓ test 2 - bar
✓ test 3 - baz

Oh noes the renderer process crashed... we will attempt to recover

...Restarting tests at 'test 4 - quux'

✓ test 4 - quux
✓ test 5 - ipsum

Taking this a step further, we are starting to see several patterns emerge with how and why renderer processes crash - it is almost always related to extremely long test runs in a memory starved environment (such as Docker).

It may even be a good idea for us to always preemptively "break up" headless runs by spec file.

In other words, we could have an option like restartBrowserBetweenSpecFiles: true which would automatically kill the renderer / browser process before moving on to a different spec file (but still rebuild the state of the UI correctly, and still have a single contiguous video recording).

To the user it would look like nothing is really different, but internally the renderer process would be killed and then restarted.

This would forcefully purge primed memory from the process, which could keep environments like docker from ever crashing to begin with.

Depends on: #6170

[jheijkoop]

We actually have these crashes halfway in a single spec and we have stalling too. I tried debugging this with strace and it seems to be constantly trying to acquire some locks.
Our app seems to make the browser allocate 400+ MB of memory fast and the whole suite can go up to 2 GB...
So resetting between specs might not be enough. Maybe between it/test is also an option?

Setting the --ipc=host does fix this, but I wonder what happens if two instances of the test run simultaneously. Could a clash occur?

[brian-mann]

How could two instances of the test run occur simultaneously? If you wanted to parallelize you would do it over two different docker containers.

[jheijkoop]

yes, two docker instances. It might be a false fear of a clash. I'm completely unaware/ignorant of what the two docker instances do share with --ipc=host

[muslim-niche]

Hi I am running test cases on aws ec2 small instance and I am having this issue
https://on.cypress.io/renderer-process-crashed
Is there any way to avoid this

[jheijkoop]

Did you try the --ipc=host fix?

[muslim-niche]

But I am not using Docker

[jheijkoop]

if not sandboxed, you might have multiple chrome instances fighting over resources. What is your setup? any concurrency? are you open to a different setup?

[khiettran]

Any update on this? As Im now getting the error with Chromium usually crashes when running amount of test suites.

[tizmagik]

We've started hitting this fairly frequently now too

[dsherret]

I'm having this happen randomly on travis-ci with cypress 3.0.2 (I just recently started using cypress so no clue if it happened in a previous version). It might be good to add this flag even with #681 resolved.

Edit: I was able to resolve my issue by only calling .visit() once and resetting the state of the application between tests. I know that's not ideal, but it works for now.

[jheijkoop]

In hindsight my fix with --ipc=host might be related to the shared memory issue I described in #350 and giving the container more shared memory might resolve crashes.

[jdtzmn]

I'm also getting this issue now with cypress v3.1.0. Any updates?

[mechanical-turk]

Hi cypress team!

We are also getting this error when we use cypress run as well as cypress open

We noticed that it happens more when we use cy.wait. We can consistently reproduce it when we use cy.wait with a value greater than 20000. This is on our circle-ci linux containers fyi.

[gwaihir8]

Hi, I'm currently trying to use cypress in Gitlab CI. I Figured out most parts, except the browser crashing.

my current gitlab CI test job is the following:

test_dev:
  only:
    - dev
  stage: test
  image: cypress/base:10
  script:
    - npm i --save-dev cypress
    - $(npm bin)/cypress run --reporter junit --reporter-options "mochaFile=results_[hash].xml,toConsole=true"
  artifacts:
    paths:  
      - cypress/videos
    reports:
      junit: results_*.xml
    expire_in: 1 week

This works great when the browser doesn't crash, including test reporting in gitlab's merge requests. However, it fails 50% of the times. Using the --ipc=host tag is afaik not an option in Gitlab CI.

[jheijkoop]

Have you tried increasing the shared memory instead, like I discribe in #350 ?

[gwaihir8]

I am using shared runners on gitlab ci, and shm-size doesn't seem to be an option for shared runners. Thanks anyway

[jheijkoop]

I think you can configure it using this documentation https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-docker-section

[egucciar]

Hi Please provide fix / explanation of this issue. It is always happening on one test case (and only one test case). I do not think it has to do with memory but there is no way to know. I was able to reproduce it locally without docker. I think it has to do with origin (subdomain) changes. Thanks

EDIT: Just ran in debug mode there is no way of knowing what is causing this problem unfortunately

[ajcann]

Hi, we're also experiencing this issue in Kubernetes (using Jenkins as our CI engine). Would be happy to provide additional information if helpful.

[jpike88]

I've recently started running into the issue, as our codebase starts to acquire more dependencies. It's intermittent and unpredictable. Sometimes I get a passing test, sometimes it fails the moment it begins.

After more experimentation, I've found that using the cypress/browsers:chrome69 image instead of the cypress/base:10 made the issue go away. This issue is likely to be tied to an older version of electron being unable to handle a larger codebase, and I think more effort should go into updating electron.

[sesamechicken]

@sesamechicken Thanks for your patience. A few things came up between then and now. It's still an issue on 6.4.0.

Platform: Docker cypress/browsers:node12.16.2-chrome81-ff75
Cypress: 6.4.0
OS: see Docker image above

Awesome. Do you happen to have a repo with an example test to run to reproduce this, or, is it with any test inside of this image?

[Oumayma-KH]

After updating to the newest version of Cypress 9.7.0, we started having the crush while running our tests in a Docker container, any updates about a solution for that issur? Thanks!

[mattvb91]

Just tried it with 10.0.0 and it cant even get as far as loading the tests before crashing

> cypress run --browser chrome

The Test Runner unexpectedly exited via a exit event with signal SIGILL

Please search Cypress documentation for possible solutions:

https://on.cypress.io/

Check if there is a GitHub issue describing this crash:

https://github.com/cypress-io/cypress/issues

Consider opening a new issue.

----------

Platform: linux-x64 (Debian - 10.11)
Cypress Version: 10.0.0

[toddbascombe]

These are the steps i follow to solve this issue:

1. I add this code to my cypress plugin folder:

    if (browser.name === 'chrome') {
      launchOptions.args.push('--disable-dev-shm-usage')
      return launchOptions
    }
    return launchOptions
  }) 

2. i use circle ci for CI/CD. In the jobs section of the cypress test in the config.yml, you add this docker image:

   docker:
       - image: cypress/included:9.4.1

• For more info on what the docker image above https://github.com/cypress-io/cypress-docker-images#cypress-docker-images-

3. In my packag.json, i updated my cypres:run script to cypress run --browser chrome
4. if your using circle ci, use the recommended cypress orbs

[Oumayma-KH]

Thank you for your response ! I will try and see if it works

[mattvb91]

These are the steps i follow to solve this issue:

1. I add this code to my cypress plugin folder:

    if (browser.name === 'chrome') {
      launchOptions.args.push('--disable-dev-shm-usage')
      return launchOptions
    }
    return launchOptions
  }) 

2. i use circle ci for CI/CD. In the jobs section of the cypress test in the config.yml, you add this docker image:

   docker:
       - image: cypress/included:9.4.1

• For more info on what the docker image above https://github.com/cypress-io/cypress-docker-images#cypress-docker-images-

3. In my packag.json, i updated my cypres:run script to cypress run --browser chrome
4. if your using circle ci, use the recommended cypress orbs

Still no luck even with this.

Interesting tho is ive put a console log before if (browser.name === 'chrome') { and it doesnt even reach that before crashing. So the process is already killed before it gets as far as even loading the config.

[estefafdez]

any update on this issue? we are experiencing it on version 11.2.0. Thanks!

[LeohsPaixao]

Browser crashed with me on version 11.2.0, any update ?

[alesiaviktorchyk]

Hey @brian-mann. Any update on this issue? We are getting We detected that the Chromium Renderer process just crashed almost every time with Cypress 11.2.0. Though everything was ok with Cypress 9.6.0.

[lakshikawaruni100]

Experiencing the same issue with 12.1.0. for almost all the tests

[mi4i0]

Also started to be experienced chrome crashes in headless mode.
It started to appear after moving from 8.4.0 to Cypress 10.11.0 (moved with one-time migration without migrating to v.9...)
We are running our tests in parallel on machines with 2 cores and 4 Gb RAM
SemaphoreCI as CI service
NodeJS 18.4
Chrome
I found that Cypress somehow had eaten all SWP and RAM memory and crashed right after (second terminal).

Is it any way to check by what Cypress process SWP and RAM were eaten?

[laerteneto]

I am also facing the same issue with Cypress 12.3.0 in a Jenkins setup.

[mattvb91]

@laerteneto can you try 9.7.0

[mattvb91]

Issue is now fixed for me on 12.3.0. I was stuck on 9.7.0 trying to upgrade every version and it never worked. This version for some reason works again. I am using this docker image with it:

# FROM cypress/browsers:node17.8.0-chrome99-ff97-slim
# https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node17.8.0-chrome99-ff97-slim/Dockerfile

FROM cypress/base:18.12.1

USER root

RUN node --version

# Chrome dependencies
RUN apt-get update && \
  apt-get install -y \
  fonts-liberation \
  libcurl4 \
  libcurl3-gnutls \
  libcurl3-nss \
  xdg-utils \
  wget \
  curl \
  # clean up
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean

# install libappindicator3-1 - not included with Debian 11
RUN wget --no-verbose /usr/src/libappindicator3-1_0.4.92-7_amd64.deb "http://ftp.us.debian.org/debian/pool/main/liba/libappindicator/libappindicator3-1_0.4.92-7_amd64.deb" && \
  dpkg -i /usr/src/libappindicator3-1_0.4.92-7_amd64.deb ; \
  apt-get install -f -y && \
  rm -f /usr/src/libappindicator3-1_0.4.92-7_amd64.deb

# install Chrome browser
RUN wget --no-verbose -O /usr/src/google-chrome-stable_current_amd64.deb "http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_108.0.5359.94-1_amd64.deb" && \
  dpkg -i /usr/src/google-chrome-stable_current_amd64.deb ; \
  apt-get install -f -y && \
  rm -f /usr/src/google-chrome-stable_current_amd64.deb

# "fake" dbus address to prevent errors
# https://github.com/SeleniumHQ/docker-selenium/issues/87
ENV DBUS_SESSION_BUS_ADDRESS=/dev/null

# a few environment variables to make NPM installs easier
# good colors for most applications
ENV TERM=xterm
# avoid million NPM install messages
ENV npm_config_loglevel=warn
# allow installing when the main user is root
ENV npm_config_unsafe_perm=true

ENV CYPRESS_CACHE_FOLDER=/root/.cache/Cypress

RUN npm i -g [email protected] [email protected] && \
    cypress install && cypress verify && cypress info && \
    npm cache clean --force && \
    apt-get install -y tar && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# give every user read access to the "/root" folder where the binary is cached
# we really only need to worry about the top folder, fortunately
RUN ls -la /root
RUN chmod 755 /root
RUN chmod -R 755 /root/.cache/Cypress
RUN mkdir -p /root/.config/Cypress/cy/production/proxy
RUN chmod -R 777 /root/.config/Cypress

# versions of local tools
RUN echo  " node version:    $(node -v) \n" \
  "npm version:     $(npm -v) \n" \
  "yarn version:    $(yarn -v) \n" \
  "debian version:  $(cat /etc/debian_version) \n" \
  "Chrome version:  $(google-chrome --version) \n" \
  "Firefox version: $(firefox --version) \n" \
  "Edge version:    $(edge --version) \n" \
  "git version:     $(git --version) \n" \
  "whoami:          $(whoami) \n"

[Neoxrus86]

Issue is now fixed for me on 12.3.0. I was stuck on 9.7.0 trying to upgrade every version and it never worked. This version for some reason works again. I am using this docker image with it:

# FROM cypress/browsers:node17.8.0-chrome99-ff97-slim
# https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node17.8.0-chrome99-ff97-slim/Dockerfile

FROM cypress/base:18.12.1

USER root

RUN node --version

# Chrome dependencies
RUN apt-get update && \
  apt-get install -y \
  fonts-liberation \
  libcurl4 \
  libcurl3-gnutls \
  libcurl3-nss \
  xdg-utils \
  wget \
  curl \
  # clean up
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean

# install libappindicator3-1 - not included with Debian 11
RUN wget --no-verbose /usr/src/libappindicator3-1_0.4.92-7_amd64.deb "http://ftp.us.debian.org/debian/pool/main/liba/libappindicator/libappindicator3-1_0.4.92-7_amd64.deb" && \
  dpkg -i /usr/src/libappindicator3-1_0.4.92-7_amd64.deb ; \
  apt-get install -f -y && \
  rm -f /usr/src/libappindicator3-1_0.4.92-7_amd64.deb

# install Chrome browser
RUN wget --no-verbose -O /usr/src/google-chrome-stable_current_amd64.deb "http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_108.0.5359.94-1_amd64.deb" && \
  dpkg -i /usr/src/google-chrome-stable_current_amd64.deb ; \
  apt-get install -f -y && \
  rm -f /usr/src/google-chrome-stable_current_amd64.deb

# "fake" dbus address to prevent errors
# https://github.com/SeleniumHQ/docker-selenium/issues/87
ENV DBUS_SESSION_BUS_ADDRESS=/dev/null

# a few environment variables to make NPM installs easier
# good colors for most applications
ENV TERM=xterm
# avoid million NPM install messages
ENV npm_config_loglevel=warn
# allow installing when the main user is root
ENV npm_config_unsafe_perm=true

ENV CYPRESS_CACHE_FOLDER=/root/.cache/Cypress

RUN npm i -g [email protected] [email protected] && \
    cypress install && cypress verify && cypress info && \
    npm cache clean --force && \
    apt-get install -y tar && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# give every user read access to the "/root" folder where the binary is cached
# we really only need to worry about the top folder, fortunately
RUN ls -la /root
RUN chmod 755 /root
RUN chmod -R 755 /root/.cache/Cypress
RUN mkdir -p /root/.config/Cypress/cy/production/proxy
RUN chmod -R 777 /root/.config/Cypress

# versions of local tools
RUN echo  " node version:    $(node -v) \n" \
  "npm version:     $(npm -v) \n" \
  "yarn version:    $(yarn -v) \n" \
  "debian version:  $(cat /etc/debian_version) \n" \
  "Chrome version:  $(google-chrome --version) \n" \
  "Firefox version: $(firefox --version) \n" \
  "Edge version:    $(edge --version) \n" \
  "git version:     $(git --version) \n" \
  "whoami:          $(whoami) \n"

@mattvb91, Hello! Can you please tell me what commands do you use in package.json to run tests?
I now start up like this:
Dockerfile contains entrypoint.sh file

entrypoint.sh contains the following code:

In package.json I use commands like the following:

I have little understanding of Docker and programming in general. I will be very grateful for your help.

[mattvb91]

@Neoxrus86 I just have package.json:

  "scripts": {
    "cy:run": "cypress run --browser chrome"
  }

and then in my github actions I mount the folder in with all of my tests and run the npm command above:

docker run --add-host="test.cypress:172.17.0.1" --network host --env-file <(env | grep cypress_) -v $(pwd):/app {{MY_CYPRESS_IMAGE_NAME}} /bin/bash -c "cd app/cypress && npm i && npm run cy:run"

in my case I dont have an entrypoint.sh and run tests on test.cypress because I have a multitenant SAAS app that has subdomains so I add a host in there that I can work off.

[Neoxrus86]

@mattvb91, Thanks a lot! I'll explore this topic in more detail.

[yaroslav-shpylyk]

I'm having this issue with electron right now Cypress ^12.3.0. Chrome works fine but I'm tied up with electron on project. For some reason it crashes after 2-3 seconds. Nothing from the list helps:

1. experimentalMemoryManagement: true
2. numTestsKeptInMemory: 0/1

on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.name === "electron") {
        launchOptions.args["disable-dev-shm-usage"] = true;
    }

    return launchOptions;
});

It fails in simple e2e test:

describe("App", () => {
    beforeEach(() => {
        cy.visit("/");
    });

    it("redirects to signup page", () => {
        cy.location("pathname").should("equal", "/signup");
    });
});

[mirobo]

Is there any progress on @brian-mann 's original suggestion?
See also my comment in the linked issue #3602 (comment)

Ideally Chromium Renderer process crashes would be

1. reported in mocha (wasn't the case in 10.7.0)
2. retried (as many times as defined in the "runMode.retries" setting)

[nagash77]

Hi @yaroslav-shpylyk can you please open a new issue with your reproducible example? This issue is quite old so we would need more recent examples to be able to properly diagnose and debug. I am closing this issue as Won't Fix, but if you are still seeing a similar problem please do open a new issue and we will be happy to investigate.