allowConsoleError should capture async errors

[lannka]

The following code illustrate the problem, where we expect a console error, but it actually might leak to outside of this test.

it('abc', () => {
  allowConsoleError(() => {
     asyncFuncThatPrintsConsoleError();
  });
})

function asyncFuncThatPrintsConsoleError() {
  setTimeout(() => {
     console.error('errr!!!')
  }, 1);
}

[lannka]

one idea is to make allowConsoleError more like expectConsoleError, which takes a regex pattern for expected console error msg. It returns a promise that resolves only when such a console error is found, or timed out.

it('abc', () => {
  asyncFuncThatPrintsConsoleError();
  return expectConsoleError('errr!!!');
})

[rsimha]

@lannka Thanks for bringing this up.

In your suggested fix, what happens if a test that generates (sync or async) console errors simply fails to call expectConsoleError? We don't want the test to just pass (this was the original problem we were trying to fix). What mechanism will cause it to fail?

[lannka]

In your suggested fix, what happens if a test that generates (sync or async) console errors simply fails to call expectConsoleError?

We still stub console.error, and capture every message in a queue. the expectConsoleError() call removes message from the queue. We fail the test if we have a non-empty queue at the end.

[rsimha]

Hmm. I want to make sure this doesn't place an undue burden on each test to explicitly declare which errors it expects twice.

So a test like this...

expect(() => {
  funcThatCallsConsoleError();
}).to.throw('Some error message');

... can be written like this today...

allowConsoleError(() => {
  expect(() => {
    funcThatCallsConsoleError();
  }).to.throw('Some error message');
});

... but with the new suggestion, might have to be re-written like this...

expect(() => {
  funcThatCallsConsoleError();
}).to.throw('Some error message');
expectConsoleError('Some error message');

I'll dig in to this some more and see if we can get the best of both worlds, where existing usage of allowConsoleError for sync errors is preserved, but async use can be supported with something like expectAsyncConsoleError.

[rsimha]

For a temporary workaround, see what #15606 does

[rsimha]

I ended up implementing more or less what @lannka suggested in #15621. Here is a usage example:

it('abc', () => {
  expectAsyncConsoleError('Foo');
  expectAsyncConsoleError('Bar');
  func();
})

If func() doesn't result in calls to console.error('Foo') and console.error('Bar'), or if it does result in a call to console.error('Baz), the test will fail.