Uncaught Exception not failing a test #1385

alurim · 2018-02-28T00:07:01Z

Operating System: Mac
Cypress Version: 2.0.4
Browser Version: Chrome Version 64.0.3282.186

Is this a Feature or Bug?

Bug... I think?

Current behavior:

Added a setTimeout(() => { throw new Error('Whoops!') }, 200) in my application code and ran a test that took a couple of seconds to run and it successfully completed.

Desired behavior:

I was under the impression that any uncaught exception will cause any test that's running to fail. If I was wrong is there any way to ensure that the test fails? I tried adding the following code but the debugger never fired:

    cy.on('uncaught:exception', (err, runnable) => {
      debugger
    })
    cy.on('window:alert', (err, runnable) => {
      debugger
    })

How to reproduce:

Add setTimeout(() => { throw new Error('Whoops!') }, 200) to any application code
Run a test that take a couple of seconds to ensure that error gets thrown
See the test succeed instead of fail.

Additional Info (images, stack traces, etc)

The text was updated successfully, but these errors were encountered:

brian-mann · 2018-02-28T00:58:10Z

Cypress will absolutely catch uncaught exceptions and fail the current test.

There are lots of tests around this behavior. What I imagine is happening here is some kind of edge case either that we didn't consider or that this isn't a real uncaught exception.

We will not catch errors if:

The test has completed (which means there is no test to associate to)
The error was thrown in an iframe or anything other than the window Cypress is bound to
It's not really an uncaught exception - where it gets handled by your app and then console.error(err) which makes it look like its uncaught but its really not.

What we'd need you to do is reduce down your example to whatever is the simplest to recreate. For instance I can see you are using HMR. If HMR is configured to work through an iframe, that could be the source.

You can look at our own tests as reference. To test uncaught exception handling - just throw an error directly in an <script> tag inside of an <html> file that you use with cy.visit(). You don't even need anything else.

Since uncaught exceptions can handle in a ton of different ways, we wrote tests that express them in all kinds of circumstances.

For instance they could happen:

in your spec code
in your app code
synchronously outside of a test
asynchronously outside of a test
synchronously in a test
asynchronously in a test
synchronously in a hook
asynchronously in a hook

All of this behavior is tested in the driver code and in the e2e tests that run Cypress in a 'pure' form the same as you run your projects.

brian-mann · 2018-04-17T18:19:57Z

Closing because we've not heard back and nobody has provided a reproducible case.

alexindigo · 2018-08-25T23:01:50Z

@brian-mann I have something similar, there is JS error happening when I do cy.go('back'),
and cypress somehow ignores it, and continues as normal.

Is there anything else I can provide to help resolve the issue?

Thank you.

jennifer-shehane · 2018-10-20T18:19:17Z

@alexindigo Hey, giving us a reproducible issue is the best way to help resolve an issue.

andezzat · 2018-12-03T03:28:17Z

Reproducible & super simplified version right here: #2487 (comment)

Someone please give it a look, because this issue just keeps happening in our app that we test in Cypress and it's causing me so much pain :(

daniel-kun · 2019-08-18T09:15:48Z

Hey @jennifer-shehane, I have a reproducible, very simple test case here:

describe("Visiting IFTTT", function() {
    it("should ignore the occuring JavaScript error", function() {
        cy.viewport(1024, 880);

        cy.on('uncaught:exception', (err, runnable) => {
            // returning false here prevents Cypress from
            // failing the test
            console.log("Catched exception");
            return false;
        });
        cy.visit("https://ifttt.com/");
        cy.get('a.sign-in').click();
    });
});

The result is (currently, it might change if IFTTT works on their site, of course) nearly 100% that there will be a JavaScript error that lets the test fail, although there's a cy.on('uncaught:exception'... there. With Cypress.on... I have the same results.

Here is a video of the failure:
https://1drv.ms/v/s!AiPF014W0FYfv8cXGBXxJ8PyToPHHw?e=cHhzsd

I'm using Cypress 3.4.0 on Windows, installed via npm install and run via .\node_modues\.bin\cypress.

Hope this helps to pinpoint the issue.

Colmea · 2019-09-24T08:49:33Z

Hi,

@brian-mann Just to be clear, when you say:

We will not catch errors if [...] The test has completed (which means there is no test to associate to)

it means that it will not work (or be flaky) if the cy.on('uncaught:exception'... is the last line of the test ?

Example:

// This action will trigger an Exception
cy.get('#myButton').click();

// Catch Exception AFTER the actual action
cy.on('uncaught:exception', (err, runnable) => {
  return false;
});

This won't work ?

swathisham · 2019-12-28T00:16:31Z

any update on this issue?

andezzat · 2019-12-28T07:16:54Z

@swathisham

#2487 (comment)

I'm tracking this issue instead ☝️

swathisham · 2019-12-29T05:24:39Z

I am getting assertion error. But test passed even though it shows assertion error

jennifer-shehane · 2019-12-31T06:34:50Z

Please follow this issue or open a new issue with a completely reproducible example #2487

brian-mann added the stage: needs information Not enough info to reproduce the issue label Feb 28, 2018

brian-mann mentioned this issue Apr 17, 2018

Tests fail because of exception in console #1565

Closed

brian-mann closed this as completed Apr 17, 2018

jennifer-shehane mentioned this issue Apr 18, 2018

Cypress failing after uncaught:exception thrown from 3rd party, even thought 'uncaught:exception' handler is listening. #987

Closed

jennifer-shehane removed stage: needs information Not enough info to reproduce the issue labels Jan 16, 2019

cypress-io locked and limited conversation to collaborators Dec 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uncaught Exception not failing a test #1385

Uncaught Exception not failing a test #1385

alurim commented Feb 28, 2018 •

edited by jennifer-shehane

Loading

brian-mann commented Feb 28, 2018 •

edited

Loading

brian-mann commented Apr 17, 2018

alexindigo commented Aug 25, 2018

jennifer-shehane commented Oct 20, 2018

andezzat commented Dec 3, 2018

daniel-kun commented Aug 18, 2019

Colmea commented Sep 24, 2019 •

edited by jennifer-shehane

Loading

swathisham commented Dec 28, 2019

andezzat commented Dec 28, 2019

swathisham commented Dec 29, 2019 •

edited

Loading

jennifer-shehane commented Dec 31, 2019

Uncaught Exception not failing a test #1385

Uncaught Exception not failing a test #1385

Comments

alurim commented Feb 28, 2018 • edited by jennifer-shehane Loading

Is this a Feature or Bug?

Current behavior:

Desired behavior:

How to reproduce:

Additional Info (images, stack traces, etc)

brian-mann commented Feb 28, 2018 • edited Loading

brian-mann commented Apr 17, 2018

alexindigo commented Aug 25, 2018

jennifer-shehane commented Oct 20, 2018

andezzat commented Dec 3, 2018

daniel-kun commented Aug 18, 2019

Colmea commented Sep 24, 2019 • edited by jennifer-shehane Loading

swathisham commented Dec 28, 2019

andezzat commented Dec 28, 2019

swathisham commented Dec 29, 2019 • edited Loading

jennifer-shehane commented Dec 31, 2019

alurim commented Feb 28, 2018 •

edited by jennifer-shehane

Loading

brian-mann commented Feb 28, 2018 •

edited

Loading

Colmea commented Sep 24, 2019 •

edited by jennifer-shehane

Loading

swathisham commented Dec 29, 2019 •

edited

Loading