Snapshot differs and fail when running all tests vs one test at time

[ghiscoding]

Describe the bug

Migrating from Jest to Vitest (to support ESM), I see that all snapshots created by Vitest have different escaping and fail when running the entire tests but if I run them as a single filename pattern then they always pass (either by filename pattern in the CLI or via the VSCode debugger). When run one file at a time, it always pass and are equivalent to Jest snaphots (escaped with double quotes) but they are very different when running all tests at once (sometime escaped with single quotes & sometime without and sometime with double quotes).

The only thing that I can think of to bypass the problem for now would be to update all snapshots so that it passes in the CI but that is basically inversing the problem, it will then fail when run one at a time. I also think that the one at a time snapshot are the correct ones since they are equivalent to Jest before I started the migration to Vitest.

Reproduction

For a reproduction you can do a git checkout of this Lerna-Lite PR 537 and run all unit tests (follow installation instructions). I will merge the PR with the snapshot failures so that it can be inspected and I can continue with my migration to ESM. The only thing that is consistent in all of this is that it always fail when all tests are ran but always pass when run one at a time.

Also worth to know that I need to run Vitest with --no-threads since Lerna/Lerna-Lite unit tests are using process.chdir() extensively, what I wonder is why it is different when running the test one at a time, technically speaking it is supposed to be isolated because of the --no-threads so why does it fail when running all tests? Is there another flag to really isolate each tests? I tried isolate, useAtomic and singleThread but none have any effects and are irrelevant, if I read correctly, when executed with --no-threads, so... ?

Fails when running all unit tests

Always pass when running one file at a time

System Info

System:
    OS: Windows 10 10.0.19045
    CPU: (12) x64 Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz  
    Memory: 10.87 GB / 31.93 GB
  Binaries:
    Node: 16.20.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - C:\Program Files (x86)\Yarn\bin\yarn.CMD 
    npm: 7.0.11 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 111.0.5563.147
    Edge: Spartan (44.19041.1266.0), Chromium (111.0.1661.54)
  npmPackages:
    @vitest/coverage-c8: ^0.29.8 => 0.29.8 
    vite: ^4.2.1 => 4.2.1 
    vitest: ^0.29.8 => 0.29.8

Used Package Manager

pnpm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[ghiscoding]

I know Vitest uses pretty-format from Jest and I found this Jest issue which seems related, but it was closed without actions.

[Bug]: string inline snapshot diff when running one or all test

I also tried to toggle escapeString option in snapshotFormat and to my surprise it has zero effect, running with true or false ends up with the exact same snapshots... isn't that a bug? Is it because I'm running test with --no-threads?

export default defineConfig({
  test: {
    clearMocks: true,
    threads: false,
    snapshotFormat: {
      escapeString: false,            // <<-- toggling this option has zero effect on shapshot format output, why???
      printBasicPrototype: false,
    },
  },
});

[sheremet-va]

All tests with the same environment in --no-threads runs in a single child_process and they all share the same global scope. Is it possible that one test just affects the other?

[ghiscoding]

I don't think that I have problem with the global scope being shared in this case. The issue is that the escaping of the snapshot is different when run one by one than it is when running all tests. I didn't have that issue with Jest, it's only in Vitest. Why would the property escape be different when run one at a time or when running all tests, why does it uses single quote vs double quotes?

[ghiscoding]

It does seem to be related to some global scope of some kind, I have to use cache: false to make sure that the unit tests are consistent in the CI. I still have a few files that use different escaping when run one file at a time or all file at once (basically the same problem since I've opened the issue).

What's strange also is that even though I use clearMock in the global config, I still had to go and add some vi.clearAllMocks() in some tests to get working one file at a time.

export default defineConfig({
  test: {
    cache: false,
    clearMocks: true,
    environment: 'node',
    testTimeout: 60000,
    setupFiles: ['./vitest/vitest-setup.ts', './vitest/silence-logging.ts', './helpers/npm/set-npm-userconfig.ts'],
    watch: false,
      provider: 'c8',
    }
  },
});

for now my only way to get consistent tests is to always run all tests and not one file at a time because some of them fails. It's too bad because that also mean that I can't always debug in VSCode one file at a time and that problem is only for snapshot escaping, the rest is working fine

[milon27]

facing same issue. my test single database is conflicting insert by 2 test files.
is there any way to run test files one after another?

[ghiscoding]

All tests with the same environment in --no-threads runs in a single child_process and they all share the same global scope. Is it possible that one test just affects the other?

I took another look at this and this comment helped me find my problem, that I was having, the root cause was that I use different Snapshot Serializers in different unit tests and when running --no-thread since they all share global scope then the result was different when ran as a single test or when run all at once...

In summary, adding the serializer to all unit tests, not just some of them but all of them, fixed my issue. The shared scope is still a bit of an issue but at least now my result is the same in all cases, so I can close this issue