EMFILE error when using fs.watch on many items

[TrevorBurnham]

Test case:

var fs = require('fs');

for (var i = 1; i <= 500; i++) {
  fs.writeFileSync("" + i + ".tmp", '');
}

for (i = 1; i <= 500; i++) {
  try {
    fs.watch("" + i + ".tmp", function() {});
  } catch (e) {
    console.log(i);
    process.exit(1);
  }
}

With Node 0.6.2 under OS X, I consistently get output of 250, meaning that 249 files were successfully watched, but an EMFILE error occurred when trying to watch the 250th. The pertinent part of the stack trace is

Error: watch EMFILE
    at errnoException (fs.js:605:11)
    at FSWatcher.start (fs.js:632:11)
    at Object.watch (fs.js:660:11)

Several folks have experienced the error when using CoffeeScript's watch mode, particularly in a directory that's a git repository (see CoffeeScript issue #1537).

Adding a 100ms interval between each file, I get the same result. So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit? Pinging @bnoordhuis.

[bnoordhuis]

So it seems that there's a hard limit on how many files one can watch simultaneously from a Node process on some systems. Is there some way that we can increase or avoid this limit?

Yes, ulimit -n 16384. It's not a Node limitation, it's a per-process limit that's enforced by the operating system (presumably OS X in your case, that's the only OS I know of that defaults to 256).

[TrevorBurnham]

Ah, thanks for the clarification.

[wombleton]

I'm running node 0.6.9 on linux mint 12. ulimit -n returns 65536 so as far as I can see it's not a file handle limit. The above test fails at 116.

Replacing fs.watch with fs.watchFile works fine.

[bnoordhuis]

@wombleton: What does the strace log look like?

[wombleton]

Error: watch EMFILE
    at errnoException (fs.js:636:11)
    at FSWatcher.start (fs.js:663:11)
    at Object.watch (fs.js:691:11)

[bnoordhuis]

Heh, that's not quite what I mean. Can you gist the output of strace node script.js? If you pass -o trace.log to strace, it'll write to a file instead of the console.

[wombleton]

Ah! Now I know about strace, I had it pegged as a slightly strange contraction. Apologies.

https://gist.github.com/1707965

[bnoordhuis]

You're running into a linux-specific system limit, namely the maximum number of inotify instances.

$ sysctl -a | grep fs.inotify
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.inotify.max_queued_events = 16384

It's something that we can work around. I've opened a libuv issue for it: joyent/libuv#300.

[bnoordhuis]

Fixed in joyent/libuv@d3efefb.

[kdvolder]

I'm using fs.watch on node 0.8.2 to watch a directory tree. I'm only watching the directories/subdirectories not each individual file.

This is working beatifully on Linux (Ubuntu 10.04). But we are hitting EMFILE error on Mac OS. It looks like this error condition is documented here:

https://developer.apple.com/library/mac/#documentation/Darwin/Reference/Manpages/man2/kqueue.2.html

as happening when

   [EMFILE]           The per-process descriptor table is full.

Our application does need to work on Mac OS as well as Unix and Windows. So I wonder... is there a way to increase this 'per-process descriptor table' size to something reasonable for our node process?

Any suggestions to work around this Mac OS limit welcome.

[kdvolder]

PS: I did try a couple of google searches to find the answer to my question, but the most useful result that turned up so far has been this issue :-)

[bnoordhuis]

@kdvolder Try e.g. ulimit -n 65536. With OS X you may have to log in again after that.

[kdvolder]

Further note: ulimit command as suggested above doesn't seem to exist on Mac OS. We tried using the equivalent Mac OS command but it didn't seem to work (my guess is because the limit we are hitting is not that of 'open files' but the 'The per-process descriptor table is full.' whatever that is, seems like it is maybe a different limit.

[kdvolder]

FYI: command we tried on OS X was this one:
sudo sysctl -w kern.maxfilesperproc=400000

Setting it however seemed not have any effect on the limit for fs.watch. Still hitting error at around entry number #249.

[kdvolder]

FYI: Decided to resolved this by not using fs.watch on Mac OS. When we detect Mac OS we switch to a polling mechanims. I'd much prefer to use fs.watch of course. But I have not found any way so far to raise the limit. (And even if I could find the magic sudo command, it would be a poor user experience if a user would have to run some 'fix my os' script as sudo before being able to use our stuff.

I do understand this isn't a node problem perse, but some type of hard OS limit and so perhaps not a reason to re-open this issue... But maybe there is something that can be done on the node implementation anyway (after all bnoordhus was able to do something to the inotify implementation of fs.watch that seems to work quite nicely on Linux. So I'm hoping someone might come along and do something similar for the Mac OS / kqueue implementation.

[medikoo]

It would be good to have that limit handy under some process property, so we can be aware, that on current platform we have following descriptors limit, and do some workarounds.

[bnoordhuis]

@medikoo That won't happen because the file descriptor limit is a platform specific thing. Non-CRT Windows doesn't really have one, for example. You're free to write an add-on that wraps getrlimit(RLIMIT_NOFILE) though.

[h2stein]

I had the same problem (256 file limit on Mac OS X) and neither of the above worked. However, I found a solution on Superuser.com: http://superuser.com/a/514049

echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 4096' | sudo tee -a /etc/profile

After rebooting, everything worked fine.

[medikoo]

Some time ago I came up with watch wrapper that uses fs.watch until descriptors limit is approached and then switches internally to fs.watchFile see: https://github.com/medikoo/fs2/blob/master/README.md#watchpath

It works really well on my side. I use it to watch thousands of files at same time on OSX.

[liukun]

A command is quite useful when checking who is consuming inotify resources:

find /proc/*/fd -type l -lname '*inotify' -print 2>/dev/null \
  | cut -d / -f 3 | (while read pid; do ps --no-headers -p $pid -o user,command; done) \
  | sort | uniq -c

Saw it at http://dev.nethserver.org/issues/2850

[brennanMKE]

This problem was not easy to fix with macOS Sierra because of a few changes which make recent instructions no longer value, specifically ulimit commands would not fix the problem as the new setting would not be persisted. I wrote up instructions for what I did to fix it for good. Now a React Native project is able to build and run the server which uses the watch utility to monitor for changes. The error for this issue what I was seeing previously and with these commands I was able to make it work.

https://gist.github.com/brennanMKE/f6aa55b452ecda2f4c7a379e21647c88