fix(runner): async assertion auto await should timeout

[hi-ogawa]

Description

• closes Test with expect(promise).resolves with promise that never resolves and without await hangs #6386

Currently documentation indicates resolves/rejects needs to be manually awaited https://vitest.dev/api/expect.html#resolves, but "auto await" mechanism is added along with toMatchFileSnapshot #3116. It looks like the intent is to make "sync" API, so the documentation is outdated, but I'm not sure if we should encourage this "sync" API because it doesn't feel really intuitive anyway.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	ed3a95b
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/66d1117e2a74a70008b45d4c
😎 Deploy Preview	https://deploy-preview-6391--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[hi-ogawa]

This came out when pnpm build, so I committed here.

[hi-ogawa]

I'm not sure but I needed to add this filter for fails.test.ts snapshot since test/cli/fixtures/fails/unhandled.test.ts's stacktrace moved a little and now it points to here

stderr | unhandled.test.ts > unhandled exception
Error: Uncaught [Error: some error]
    ...
    at file:///home/hiroshi/code/others/vitest/packages/runner/dist/index.js:146:14
    at file:///home/hiroshi/code/others/vitest/packages/runner/dist/index.js:529:11
    at runWithTimeout (file:///home/hiroshi/code/others/vitest/packages/runner/dist/index.js:61:7) Error: some error
    at /home/hiroshi/code/others/vitest/test/cli/fixtures/fails/unhandled.test.ts:8:11
    ...

[hi-ogawa]

Previously it was waiting for Promise.allSettled, but I think we can simply use Promise.all since just throwing on first rejected promise would match with how test works.

[sheremet-va]

Not really, failTask will fail with every error, not just the first one

[hi-ogawa]

Okay, I copied Promise.allSettled.

[hi-ogawa]

Errors are doubled since they also treated as unhandled rejections. This is the entire test output:

 ❯ test/async-assertion.test.ts (1)
   × multiple errors

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  test/async-assertion.test.ts > multiple errors
AssertionError: expected 'xx' to be 'yy' // Object.is equality

Expected: "yy"
Received: "xx"

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯

 FAIL  test/async-assertion.test.ts > multiple errors
AssertionError: expected 'xx' to be 'zz' // Object.is equality

Expected: "zz"
Received: "xx"

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 2 unhandled errors during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
AssertionError: expected 'xx' to be 'yy' // Object.is equality

Expected: "yy"
Received: "xx"

 ❯ ../../packages/expect/dist/index.js:1774:31
 ❯ processTicksAndRejections node:internal/process/task_queues:95:5

This error originated in "test/async-assertion.test.ts" test file. It doesn't mean the error was thrown inside the file itself, but while it was running.
The latest test that might've caused the error is "multiple errors". It might mean one of the following:
- The error was thrown, while Vitest was running this test.
- If the error occurred after the test had been completed, this was the last documented test before it was thrown.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
AssertionError: expected 'xx' to be 'zz' // Object.is equality

Expected: "zz"
Received: "xx"

 ❯ ../../packages/expect/dist/index.js:1774:31

This error originated in "test/async-assertion.test.ts" test file. It doesn't mean the error was thrown inside the file itself, but while it was running.
The latest test that might've caused the error is "test/async-assertion.test.ts". It might mean one of the following:
- The error was thrown, while Vitest was running this test.
- If the error occurred after the test had been completed, this was the last documented test before it was thrown.
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 Test Files  1 failed (1)
      Tests  1 failed (1)
     Errors  2 errors
   Start at  16:50:11
   Duration  161ms (transform 19ms, setup 0ms, collect 14ms, tests 11ms, environment 0ms, prepare 37ms)

[sheremet-va]

They should be treated as unhandled rejections tho? We are catching them

Were they treated as unhandled rejections before?

[hi-ogawa]

Yeah, this is not ideal but the behavior hasn't changed. It looks like hasn't been tested, so I just added tests.

There is a similar test here

vitest/test/core/test/jest-expect.test.ts

Lines 790 to 795 in 85fb94a

	describe('promise auto queuing', () => {
	it.fails('fails', () => {
	expect(() => new Promise((resolve, reject) => setTimeout(reject, 500)))
	.resolves
	.toBe('true')
	})

but it turned out this is failing for a different reason:

TypeError: You must provide a Promise to expect() when using .resolves, not 'function'.

If I rewrite this test into what's probably intended, then this causes an unhandled rejection so the entire test fails.

[hi-ogawa]

I don't quite understand, but removing async in a seemingly unrelated place fixed it e7d5597

[hi-ogawa]

Okay, I copied Promise.allSettled.

[hi-ogawa]

I think I understand now. When returning promise from async () => { return somePromise } and somePromise rejects, even if we handle rejections of somePromise (e.g. by somePromise.catch or Promise.allSettlled([somePromise]), somePromise rejection still leaks to the promise returned from async function calls.