Ruff using 50-100gb Ram

[tomhamiltonstubber]

Hi folks,

ruff has just tried to use 100gb of memory on my machine while running the formatter twice at the same time:

I think this is related to #8147 as it was triggered by PyCharm's File Watcher functionality and that seems to be when that happens. This is for a pretty big project, not public so I can't share.

The command that caused this was ruff format, the File Watcher config is shown here:

My settings looks like:

[tool.ruff]
    line-length = 120
    exclude = [
        '*/migrations/*',
        '*_extra_messages.py'
    ]

[tool.ruff.format]
    quote-style = 'single'

[tool.ruff.lint]
    extend-select = ['I']
    ignore = ['E402']

[tool.ruff.lint.isort]
    combine-as-imports = true

and I'm using version 0.2.0

For the moment, I'm going to turn off the 'Trigger the watcher on external changes' flag as I presume this causing the issue, but lmk if you need more info.

That didn't work and it's now happening regularly. I've had to disable the File Watcher completely for now, I still can't replicate using the command line.

[MichaReiser]

That's a lot of ram...

I think this is related to #8147 as it was triggered by PyCharm's File Watcher functionality and that seems to be when that happens.

That sounds very likely because there's a bug that causes cache corruption if two processes write to the same cache file at the same time.

That didn't work and it's now happening regularly. I've had to disable the File Watcher completely for now, I still can't replicate using the command line.

Are you saying that this only happens to processes spawned by the file watcher but running the same command over CLI doesn't suffer the same problem? That would be surprising because both the file watcher started and the CLI started ruff should load the same cache.

Can you try to change your file watcher to pass --no-cache to see if the error persists? You can also try to clean your cache directory (default location is .ruff-cache)

[tomhamiltonstubber]

Are you saying that this only happens to processes spawned by the file watcher but running the same command over CLI doesn't suffer the same problem? That would be surprising because both the file watcher started and the CLI started ruff should load the same cache.

I imagine it's because of the cache corruption as you say. Using the same command in the terminal doesn't cause the issue.

Can you try to change your file watcher to pass --no-cache to see if the error persists? You can also try to clean your cache directory (default location is .ruff-cache)

I'm running the file watcher now with the flag, I'll monitor and report later on the status.

[tomhamiltonstubber]

Okay I can now cause the error with ruff check in the terminal. Caps out at 100gb ram. Running with the --no-cache flag works.

Once you do a release of the new code I'll check again

[MichaReiser]

@tomhamiltonstubber thanks for the update. That's terrifying but glad that it is related to caching. You can try to remove the .ruff-cache directory and use --no-cache in the file watcher only because it should be less likely that you run into the race when using ruff from the CLI.

Thanks for keeping us posted!

[charliermarsh]

Yeah my guess is that the cache itself is now in a bad state — I’d suggest removing it and running with no-cache for now if you’re seeing this consistently, but I believe it should be fixed in the next release.

[MichaReiser]

@tomhamiltonstubber we've released a new version that includes the fix. Does this resolve the problem for you?

[MichaReiser]

I'm closing this. @tomhamiltonstubber feel free to re-open/comment if you're still experiencing the issue when using the latest ruff release.

[tomhamiltonstubber]

@MichaReiser I've been running without the no-cache option for the last couple of days with the new version and haven't had any issues so I think it's been solved. Thanks a lot!

[MichaReiser]

Removing the -no-cache in the latest version should be okay if the caching was the root cause. But whether caching is enabled or not probably doesn't matter in your use case