Linux: implement a different watcher that does not require file handles per file in the workspace folder

[GregoryLundberg]

• VSCode Version: 1.21.0
• OS Version: Arch Linux

Steps to Reproduce:

1. Run 'code' from command line or click menu item to run it.

Does this issue occur when all extensions are disabled?: Yes

Dialog message indicates the web page has help. It does not.

Bumped (soft) open file limit from 1024 to 4096: no help.

[Stigjb]

I just got this myself when opening a new folder in a window I had open. I clicked the button to see instructions, but nothing happened.

VS Code version: 1.21.0
OS version: Linux Mint 18.2 Sonya 64-bit.

[bpasero]

@GregoryLundberg @Stigjb the button should open a browser pointing to these instructions: https://code.visualstudio.com/docs/setup/linux#_error-enospc

Did no browser open?

[GregoryLundberg]

It opened for me, but there is nothing there about running out of file handles, only about running out of disk space. Neither are true and I clicked to never see the notification again because it's a obviously a false-positive.

[bpasero]

@GregoryLundberg so you did not see https://code.visualstudio.com/docs/setup/linux#_error-enospc open in your browser? What page did open for you then?

[GregoryLundberg]

YES the browser opened. YES it showed that page. NO it does not speak about how to fix your program bug claiming it ran out of FILE handles.

ENOSPC means out of space on the drive. I am not.

The notification says it's out of FILE handles. It is not. I increased the limit from 1024 to 4096 to no effect.

As I read the web page, it's about running out of some other sort of handles so it's not about FILE handles and does not apply to the notification.

I clicked to ignore the error because it's obviously a false positive since there are plenty of file handles.

I thought perhaps I could re-enable it to post a screen shot but I don't see a settings option for it. Where did you hide the options? There doesn't look to be anything under File|Preferences|Settings or <gear>|Settings.

[GregoryLundberg]

OK I read what that page says and half a million file "watches" won't do. If it's really a problem you'll either need to clean up your code or document that VS Code is only for small to moderate sized projects.

My project tree has far more than half-a-million files, and there is no reason for you to "watch" every single one of them, which is, I guess, what that error is about.

I guess your notification meant "out of file-watch handles" and someone omitted the qualifier.

Perhaps the breakage is you're trying to watch everything when you really only need to watch files which are actually open in the editor (which, at the time the notification appeared was one: your release notes, which automatically appears).

[Stigjb]

The browser did not open for me.

[bpasero]

@GregoryLundberg we should probably update the documentation so that its clear that the ENOSPC error is in fact the same error for running out of file handles.

Currently we have no other way than requiring you to change the limit in your config file to make it work.

@Stigjb that is weird. if you have a http link in the editor and you ctrl+click it, does it also not open any browser?

[GregoryLundberg]

You're serious? What in the world do you need more than 4096 open files for? If you're trying to open every file in my project, I doubt I can set the limit that high.

I suppose I could try bumping to 32K. It won't come close to opening every file, but it sounds sufficiently insane. Not that I can tell if it has an effect because I don't know how to re-enable the notification, so it seems sort of pointless.

[bpasero]

@GregoryLundberg two things we can try. First, we have a setting "files.useExperimentalFileWatcher", can you try setting that to true, restart Code and then see if it happens still?

[GregoryLundberg]

How do I re-enable the notification? Not much point in testing if I can't see the results.

[bpasero]

@GregoryLundberg that notification shows up per workspace, so if you just rename the folder you open it should show again.

[GregoryLundberg]

Sorry. No can do.

[bpasero]

@GregoryLundberg no problemo, an easier way of doing it: from the command line run code --user-data-dir <some folder>. This will open a fresh version of Code with no prior state. That also means that you will have to change the setting I mentioned in this case again and restart once.

[GregoryLundberg]

Too bad there's no way to re-enable the notifications properly. Probably should open a new issue for that. Seems like a pretty serious design flaw.

Using a forced directory, I see the notification, again.
I switched to your experimental watcher and still see the notification.
I set the max open file handles to the limit of 4096 (can't set it higher) and still see the notification.

Are you sure the message is not incorrect and it's not about file handles at all? The documentation says file WATCH handles. Maybe the documentation is correct and the notification message is wrong?

Well, whatever is wrong, it seems the correct option is to ignore it. You don't seem to need whatever you're complaining about.

[bpasero]

For resetting the notification preference, we have #24815

The message is correct and it is about file handles. The file watcher we have needs 1 file handle per file on Linux to support the watching (unfortunately). When the list of file handles exceeds the OS limit, VS Code will still function but some file events will not be recorded and be missing. E.g. if you delete a file in the Linux explorer, you will not see that file go away from the VS Code explorer.

Another thing to try is to configure files.watcherExclude setting. The idea is that you can exclude parts of the workspace from watching that contain many files. You can do that back when you run normally and when you set back to "files.useExperimentalFileWatcher: false".

Is there a particular large folder that you hide, e.g. some out folder with compiled contents?

[GregoryLundberg]

So problem is simply bad design on your part.

Since you use one file handle per file, you have a limit on project size of 4096 files.

You should clearly document, therefore, that VS Code is only usable for small projects. Since my project has over 8000 files just for the primary C++ headers and footers, there is no way for you to work without error.

I've not used inotify(7) but would have assumed it was more efficient than you're implying.

[bpasero]

@GregoryLundberg yeah it is bad design on our part, we should only watch what is actually needed by having an API where you opt-in to specific paths that should be watched as opposed to watching everything.

I do not think there is a way to watch files on Linux recursively without requiring a file-handle per file in the workspace. On Windows and macOS however, file watchers do NOT require any file handles per file. So this issue is actually only showing up on Linux unfortunately.

[GregoryLundberg]

According to the man page, one file-watcher handle per directory should do for all files and directories within that directory. It does not say recursive, so, I'd assume (not tested) that creating, renaming or deleting a directory would get a notification event on the directory where it appears.

As I said, though, I've never actually used inotify and am not sure I could throw together a test case.

[sagebind]

I'm also noticing this. I'm much more curious to hear why this is suddenly a new problem. Why did 1.20 not complain about file handles? Was it just silently ignored? I've used Code on workspaces with 10,000+ files before with no obvious issues or loss in functionality.

[bpasero]

@sagebind yes it was silently ignored and you would see no file watching support without actually understanding what is going on.

[lucafaggianelli]

Any update on this bug?

[Belar]

I run into watcher issues a few weeks ago, seemed exactly like the problem described in #50417 which led me to this issue and its references; all of which fit the pattern with node_module, watcherExclude and so on.

Here are 2 observations which I didn't see mention anywhere, although behind the scenes (just a guess, I'm not familiar with VS Code code base) they "feel" like related to multi-root workspace (#40898)

1. Noticed with VS Code's settings.json (when opening using "Open Settings (JSON)"); Opening file from outside the root of workspace breaks watcher
2. Opening different directory in a new window breaks watcher

If those are expected behaviors, then you can skip the rest of the post, which is reproduction steps.

Test objects are 2 clean projects (App1, App2) generated with create-react-app (npx create-react-app <project_name> is all that's needed). After running @dotnetCarpenter script from #45295 (comment) we get:

MAX file watches allowed
8192
Current files in project - excluding node_modules
52
Current files in project - including node_modules
27741

Max file watchers count is the default one, and each project, without node_modules, has 52 files.

First case

App1 can be opened in VSC without problems, npm start will run just fine (file changes will be recorded by VSC and the dev mode watcher). However, after opening e.g. settings.json with > Preferences: Open Settings (JSON) familiar pop-up shows up and npm start on App2 will render watcher error (that's theoretically on 104 files - non node_modules elements from App1 and App2).
Running apps isn't necessary, a "reliable", minimal reproduction is to open App1, open src/App.js, open settings.json, do changes with saving to the src/App.js and switch tabs to settings.json - should suffice for pop-up to show up (although it's not 100%, more like 85), npm start will render watcher errors.

Second case

Open App1 in VSC, run it (order doesn't matter), changes will be recorded correctly e.g. correctly reflected in VSC's git tool. Open new VSC window (Ctrl + Shift + N), open App2. Familiar pop-up may (should, but not always) show up now, and if you try to run App2 (npm run start) there will be no watchers available.

When working in single directory and files within, the exclusion seems to be working as expected (at least I couldn't find a minimal reproduction; same when paying attention in real life projects). Also, didn't notice above issues (out of watchers when trying to run app in dev mode) with other editors.

I hope above helps at least a bit.

VS Code info at the time of writing:

• v1.29.1
• clean settings or "files.useExperimentalFileWatcher": false in global settings (which is default Linux, but in other reports it was suggested to set it explicitly)
• all extensions disabled

[ghost]

Also experiencing this on Ubuntu 18.04.

[jhanschoo]

To help with diagnosing this issue, I came up with this script that prints files being watched in a workspace or folder (remember to change the workspace variable in this zsh script)

#!/usr/bin/env zsh
# this script prints a series of paragraphs:
# - each paragraph corresponds to an inotify instance
# - the first line prints the process owning that inotify instance
# - each line in the rest of the paragraph corresponds to a file being
#   watched by the inotify instance

workspace=${HOME}/src/writerite/frontend-react

# create an associative array of inodes to their paths in the workspace
typeset -A inode_map
# list all the files in the workspace, then pass all the paths to ls to obtain
# their corresponding inodes, then sort these lines to uniquify the inodes in case
# duplicate hard links are present; then for each line
find "$workspace" -exec ls -di {} + | sort -nu -k1,1 | while read line
do
  # split it into words by the word-separator, then initialize an array with these words
  line_arr=(${=line})
  # the first word is the inode, so assign it as the key, whereas the other words form the path
  inode_map[$line_arr[1]]="$line_arr[2,-1]"
done

# for each inotify instance (found through iterating through each
# file descriptor opened by each process and seeing if it is an
# inotify file descriptor)
find /proc/*/fd/* -type l -lname 'anon_inode:inotify' 2> /dev/null |
while read line
do
  # print the process name for later printing
  process=$(sed -e 's@^/proc/\([[:digit:]]*\)/fd/[[:digit:]]*@\1@' <<< $line)
  cat "/proc/${process}/cmdline"
  echo

  # for each path to the file descriptor
  echo $line |
  # modify it to a path to the file descriptor info file
  sed -e 's@/fd/@/fdinfo/@' |
  # and print its contents
  xargs cat |
  # filter for the lines containing information about the files
  # watched in this instance (one line corresponds to one watch)
  grep inotify |
  # extract the hex-encoded inode of the file being watched
  sed -e 's/^.*ino:\([[:xdigit:]]*\) .*$/0x\1/' |
  while read line
  do
    # if the inode corresponds to a file in the workspace
    if [[ $inode_map[$((line))] ]]
    then
      # echo the file path
      echo "$inode_map[$((line))]"
    fi
  done
  echo
done

[jhanschoo]

Related to this issue, my script also catches that VSCode's TypeScript support watches node_modules files as well, even if tsconfig.json only includes a subdirectory without the node_modules folder.

[kid1412621]

@GregoryLundberg two things we can try. First, we have a setting "files.useExperimentalFileWatcher", can you try setting that to true, restart Code and then see if it happens still?

work for me! lifesaver

[GitMensch]

Was the experimental watcher activated by files.useExperimentalFileWatcher dropped or made the non-changeable default?

[cha0s]

It doesn't work. I tried files.useExperimentalFileWatcher, but the issue persists. See: #40898

[bengtj]

As this still cause lots of issues for us (large repo) after setting kernel limits to max I wonder if the Linux possibilities has been investigated properly. I'm no expert at inotify() and I agree that there is probably no recursive flag like bWatchSubtree like for Windows. But to me it looks like there could be a workaround by watching directories instead.

From "man inotify":
When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

Using this would probably mean some framework where when you want to add a file watch you would instead add an entry into the parent directory watcher that will handle forwarding for file events. But to me it sounds doable. (Not knowing vscode internals at all.)

Seems like there are existing wrappers for making inotify recursive, for example:
https://github.com/dzchoi/Inotify-wrapper-for-recursive-directories

On a related note; is there maybe already some method to get vscode to refresh directory contents whenever you expand a directory? This would be very useful to not have to remember to hit the refresh button for those folders that we are excluding from being watched. When a file is renamed or similar.

[awilkins]

From inotify(7) man page :

Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional
watches must be created. This can take a significant amount time for large directory trees.

If monitoring an entire directory subtree, and a new subdirectory is created in that tree or an existing directory is renamed into that tree, be aware that by the time you create a watch for the new subdirectory, new files (and subdirectories) may already exist inside the subdirectory. Therefore, you may want to scan the contents of the subdirectory immediately after adding the watch (and, if desired, recursively add watches for any subdirectories that it contains).

VSCode has 2 bits of code that mention inotify

• src/vs/platform/files/node/watcher/unix/chokidarWatcherService.ts
• src/vs/platform/files/node/watcher/nsfw/nsfwWatcherService.ts

The unix service (which uses chokidar) is marked deprecated (as is the WindowsWatcherService), so I presume the direction of travel (and maybe previously the "experimental" one) is nsfw.

Watcher creation is handled in diskFileSytemProvider.ts ; this confirms that the "legacy" watchers (Windows and Unix) are only used ...

• If you have polling enabled
• If you have legacy watching enabled

The setup code passes a list of folders to the watcher.

nsfw as a package (and a very cursory flick over the code) seems to be aware that the right way to watch a directory tree in Linux is to put inotify watches on the folders and not the files, and similarly is aware that you use ReadDirectoryChangesW on Windows.

What nsfw doesn't seem to be aware of that Watchman does, is that the inotify events queue will overflow if there are a lot of events*. Users with high file update rates may wish to consider deepening /proc/sys/fs/inotify/max_queued_events. Watchman catches overflows and assumes every file in that folder changed if they occur.

What also doesn't seem to be the case is that (correct me if I'm wrong here devs... ) nsfw doesn't have a facility for ignoring subdirectories. Any paths you set in files.watcherExclude[] are handled in VSCode after events return from the watcher service - this means that globs that exclude folders that would never match like **/.git/objects/** or **/node_modules/** still return events from nsfw which are then subsequently ignore, whereas you could actually just never register a inotify user watch on these folders. Whether this improves performance a great deal I don't have numbers to support, but I'd expect that marshalling big wodges of events from a C++ program to be handled by a JS program is probably less efficient than just never doing that.

My suggestion for an improvement would be to put a callback in nsfw that offers up every folder that it's intending to watch to the client code, and allow the client to cancel that watch before establishing it, or to just add an extra event type to the existing callback that does this.

---

* This assumption is based entirely on it lacking the string "overflow" in it's code tree. If

[bpasero]

We will be shipping a new file watcher (#132483) starting in insiders tomorrow that allows to exclude folders natively from watching via the files.watcherExclude setting. This has the advantage that you will not be running into file handle issues on Linux when you exclude large folders that need to be recursively watched.

I do not see a solution on Linux to watch a folder recursively without requiring a file handle per folder in the tree, so I will go ahead and close this issue. Recursive watching is a requirement of VS Code that we cannot lift because extensions rely on file events from any file within the opened workspace.