Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File watcher resulting in high CPU use on Mac/Linux for large folders #3998

Closed
bpasero opened this issue Mar 11, 2016 · 116 comments
Closed

File watcher resulting in high CPU use on Mac/Linux for large folders #3998

bpasero opened this issue Mar 11, 2016 · 116 comments
Assignees
Labels
debt Code quality issues file-watcher File watcher perf
Milestone

Comments

@bpasero
Copy link
Member

bpasero commented Mar 11, 2016

We are using chokidar for file watching. Chokidar is known for performing bad over large folders (see paulmillr/chokidar#410). To investigate:

  • can we consume a newer version of chokidar with the scalability issue fixed
  • can we change our watching strategy to only watch what we need instead of always watching root
  • can we switch to another watching library that is more efficient

Workaround:
Find the large folders in your workspace and add them under the files.watcherExclude setting

@bpasero bpasero added the bug Issue identified by VS Code Team member as probable bug label Mar 11, 2016
@bpasero bpasero self-assigned this Mar 11, 2016
@bpasero bpasero added this to the Backlog milestone Mar 11, 2016
@bpasero bpasero modified the milestones: Backlog, March 2016, April 2016 Mar 13, 2016
@paulmillr
Copy link

@bpasero i'll keep an eye on improving the scalability in chokidar to fit into the release. Let me know if you have a specific test repo (with large folders?) to check the performance.

@bpasero
Copy link
Member Author

bpasero commented Mar 15, 2016

Some good ones are TypeScript (https://github.com/Microsoft/TypeScript) and of course the full Chromium workspace is probably the largest I know: https://chromium.googlesource.com/chromium/src.git

@paulmillr
Copy link

Just checked TypeScript, it consumes around 80% of one core for ~2 seconds and then it's back to 0%.

Cloning Chromium.

@paulmillr
Copy link

Although when useFsEvents is false (non-default option on OSX), the CPU usage becomes terrible.

I'll investigate the new PR in chokidar too.

@sibelius
Copy link

is there some workarounds for now?

I'm using a macbook air with the latest version of vscode

@bpasero
Copy link
Member Author

bpasero commented May 11, 2016

@sibeliusseraphini there is a way to exclude large folders from watching in settings. It has a downside though that file events are no longer detected in those folders (e.g. the explorer might not update properly for that folder).

@sibelius
Copy link

is there a simple way of marking a folder as an excluded one? like in Webstom?

it should also be possible to invalidate the cache/file watching or restart it as well

@bpasero
Copy link
Member Author

bpasero commented May 11, 2016

Currently not. See http://code.visualstudio.com/updates/vMarch#_workbench and find Exclude files from watching

@haamis
Copy link

haamis commented Mar 11, 2021

Also ran into this issue when writing data to a gzip file, WatcherService uses about 75% CPU (a bit under one logical CPU/core). The file lives under my WSL home directory and I'm using Remote-WSL. Excluding **/*.gz (in user, remote and workspace settings) seems to do nothing even after restarting Code.

EDIT: Excluding **/** seems to fix it for me. Not an ideal solution obviously.

@jugmac00
Copy link

jugmac00 commented Mar 26, 2021

I also encountered this problem.

The workaround (files.watcherExclude... ) worked, but is pretty well hidden.

The Jetbrains IDE's has an entry in the context menu when clicking on a folder - maybe that would be a good idea for VS Code, too.

@electrovir
Copy link

electrovir commented May 6, 2021

I just had this issue suddenly start yesterday (May 5, 2021) in a 7.3 GB repo despite not having an issue for the past three years. I already have lots of properties within files.watcherExclude (hence no issue before), and only one root folder. The process sticks at 100-400% CPU usage.

@krsinghshubham
Copy link

Why don't we migrate to some other file watcher service other than chokidar. Excluding the folders will hamper the file update processes.

@ghuser
Copy link

ghuser commented Jun 9, 2021

Why don't we migrate to some other file watcher service other than chokidar. Excluding the folders will hamper the file update processes.

By debugging vscode's file-watcher code in the past I came to conclusion that this is what causes the problem.
I think it would worth it for vscode maintainers to fork that project and provide the feature, in order to avoid all that code refactoring.

@doubledgedboard
Copy link

ran into this issue on a golang project with some recursive symlink folders inside it (my mac got VERY hot, which I guess was a good thing since I knew something was wrong)

Adding the paths to files.watcherExclude helped, but it was a very non-obvious fix that only surfaced after extensive googling of CPU usage, finding the Process Explorer, and confirming the watcherService as the culprit

Version: 1.57.1
OS: Darwin x64 19.6.0

@karrtikr
Copy link
Contributor

Can we detect if it's a large folder, and decide whether we want to enable watcher accordingly?

@A-Sam
Copy link

A-Sam commented Sep 18, 2021

The "files.watcherExclude" doesn't work. A huge repo is constantly stuck at 100% all cpus even if i disabled everything:
"files.watcherExclude": {"**/**" : true}. Any other workarounds? I've done it also for search.exclude

@bpasero bpasero modified the milestones: Backlog, October 2021 Oct 11, 2021
@bpasero bpasero removed upstream Issue identified as 'upstream' component related (exists outside of VS Code) upstream-issue-linked This is an upstream issue that has been reported upstream labels Oct 11, 2021
@bpasero
Copy link
Member Author

bpasero commented Oct 11, 2021

Plan is to move to a new file watcher on all platforms that does:

  • allow to natively exclude folders via files.watcherExclude (currently the exclude is applied not very efficiently in JS code)
  • does not recursively scan the watched folder on startup [1]

[1]
On Linux, watching a workspace requires to walk the hierarchy all the way down and attach listeners on each folder on the way. But with supporting files.watcherExclude natively on the level of the watcher, this should improve drastically when you exclude large folders as needed.

Tomorrows insider build will ship with the new watcher.

@valhuber
Copy link

valhuber commented Oct 12, 2021

Version: 1.62.0-insider (Universal)
Commit: e96a974
Date: 2021-10-11T05:14:31.072Z
Electron: 13.5.1
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 20.6.0

Not sure if this is the "tomorrow's build", but just downloaded and seeing this:

That was asking VSC to monitor my entire ~ folder.  A lot.  If I narrow it to my dev work, it's pretty reasonable.... down to .1%, total cpu goes from 8% -> 2%.  So, this appears to be addressing the problem on my machine...

@bpasero bpasero added the debt Code quality issues label Oct 12, 2021
@bpasero
Copy link
Member Author

bpasero commented Oct 12, 2021

Today's insider build includes the new file watcher: https://code.visualstudio.com/insiders/

Linux users: when you configure files.watcherExclude, we currently do not support recursive glob matching for exclusions, so please configure an exclude rule that is either:

  • the full absolute path of a folder to ignore
  • the workspace relative path of a folder to ignore

We have some cleverness to convert a exclude pattern such as **/node_modules/** into a path of <workspace folder>/node_modules, so in most cases the existing exclusion rules should work unless they are more complicated than that.

@bpasero bpasero closed this as completed Oct 12, 2021
@A-Sam
Copy link

A-Sam commented Oct 12, 2021

Tested the master on Linux and this problem doesn't exist anymore.

Thanks @bpasero and the devs for the efforts it was a real headache to kill it upon every startup.

@jrmyio
Copy link

jrmyio commented Oct 26, 2021

@bpasero Not sure if this is to be expected, but 1 of the two processes, is still using 17k file inotifys for me. The other 1 is recuded down to 3k when using the insider build.

The process that is still using the 17k is started as soon as I open a typescript file within my project:
/home/test/.vscode-server-insiders/bin/dc1a6699060423b8c4d2ced736ad70195378fddf/node --max-old-space-size=3072 /home/test/.vscode-server-insiders/bin/dc1a6699060423b8c4d2ced736ad70195378fddf/extensions/node_modules/typescript/lib/tsserver.js --useInferredProjectPerProjectRoot --enableTelemetry --cancellationPipeName /tmp/vscode-typescript1000/34436af53053e9135e28/tscancellation-c3bde92247843ca61b50.tmp* --globalPlugins typescript-vscode-sh-plugin --pluginProbeLocations /home/test/.vscode-server-insiders/bin/dc1a6699060423b8c4d2ced736ad70195378fddf/extensions/typescript-language-features --locale en --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation

The one that was actually reduced from 17k to 3k is:
/home/test/.vscode-server-insiders/bin/dc1a6699060423b8c4d2ced736ad70195378fddf/node /home/test/.vscode-server-insiders/bin/dc1a6699060423b8c4d2ced736ad70195378fddf/out/bootstrap-fork --type=watcherServiceParcel

I am not sure if this is to be expected or this fixes only half of the problem?

@bpasero
Copy link
Member Author

bpasero commented Oct 27, 2021

@jrmyio please protest in this issue microsoft/TypeScript#46351

@github-actions github-actions bot locked and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
debt Code quality issues file-watcher File watcher perf
Projects
None yet
Development

No branches or pull requests