EBADF error when watching tons of files on macos

[bluwy]

Describe the bug

When watching many files on macos. spawning a child process causes EBADF errors consistently. (likely due to reaching maximum file descriptors?)

Versions (please complete the following information):

• Chokidar version: v4
• Node version: 18, 20
• OS version: MacOS 15.1

To Reproduce:

https://github.com/bluwy/chokidar-lots-of-files-repro

Steps:

1. node gen.js (generate files in src/ to watch)
2. node test.js (watches files and tries to run the ls bash command)

Expected behavior

EBADF errors should not appear when spawning a child process after watching tons of files.

Additional context

With chokidar v4 removing fsevents and using fs.watch, this issue is blocking Vite and Nuxt from upgrading to v4 as there's no easy workaround to the problem (other than usePolling).

• 6.0.0-beta.7 regression, browser does not open with "spawn EBADF" error vitejs/vite#18527
• EBADF error with Nuxt 3 and Vite on macOS, requiring esbuild version override nuxt/nuxt#29744 (has Nuxt-specific repro, which I believe relied on unstorage v1.13.0 that uses chokidar v4)
• fix: revert chokidar to v3 unjs/unstorage#502

Users have also tried adjusting ulimit to extend the maximum file descriptors, but it didn't fix the issue.

[paulmillr]

fs.watch with {recursive: true} can be used as backend. That would be breaking change (v5), because it is not available on Linux / IBM until node.js v20. It also comes with following caveats:

• "On Windows, no events will be emitted if the watched directory is moved or renamed. An EPERM error is reported when the watched directory is deleted."
• No 'ignore' option: which means if there is some huge directory which constantly changes (like build/), it will emit a lot of events

It's unclear how to solve issue number one.

[bluwy]

Re no1, I suppose the only way it to test against EPERM errors and stat the filesystem again for the new change, but you won't be able to find out the new name if it's moved I think.

Re no2, yeah that's a huge bummer to workaround of. I'm not sure why there's no similar option yet, FWIW @parcel/watcher supports that (as a fs.watch replacement itself) but I suppose that's out of scope.

About Linux / IBM recursive support, I noticed that node manually implemented it on the JS side rather than natively (source 1, source 2), so perhaps there's not much gain using recursive there and what chokidar implements now should be on par perf-wise.

[paulmillr]

v4 moved over from our own native fsevents dep to reduce pain (of using native deps). Of course, we won't be using parcel.

Reverting to fsevents usage is a possibility if there's someone trustworthy willing to maintain it. Until then it's a no-go.

[bluwy]

That's fair. Another option is to figure out a workaround on macos, maybe there's a system/bash command that could fix this like linux.

[43081j]

Just to make sure I understood correctly: the current problem then is because we implement recursive watching ourselves? So with a huge amount of files, we watch each individual one

But if we could use node's recursive, it'd probably do the same under the hood and hit the descriptor limit?

Maybe we could do some smarts around watching parents or something.

Do we know any repos where this problem happens often? Would be good to see what kind of structure we're dealing with in real world

[bluwy]

Yes, but for recursive, it only does the same under the hood for linux (JS-based). Windows and macos have native recursive watching, which should be more efficient.

Unfortunately I haven't found any repros at the moment, but I'm planning to try to create one today.

[bluwy]

Here's a repro and I'm able to reproduce the issue too: https://github.com/bluwy/chokidar-lots-of-files-repro