Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LRU auto-strategy for GC #855

Open
dmpetrov opened this issue Jul 3, 2018 · 2 comments
Open

LRU auto-strategy for GC #855

dmpetrov opened this issue Jul 3, 2018 · 2 comments
Labels
A: gc Related go garbage collection feature request Requesting a new feature p3-nice-to-have It should be done this or next sprint
Milestone

Comments

@dmpetrov
Copy link
Member

dmpetrov commented Jul 3, 2018

It would be convenient to set up a strategy like LRU instead of removing everything or by time #678.

Use cases:

  1. dvc gc --stragegy LRU 10Gb - release 10Gb from cache
  2. dvc config Core.LRU_Limit 20Gb && dvc run -d input/ -o output.tsv Rscript - it will GC automaticaly as a result of dvc run if cache size if more then 20Gb

This should work on a shared cache as well for multiple users.

It looks like this appproach will require a special state file in a cache directory to support LRU strategy and some of the DVC commands will modify the file: dvc run, dvc repro etc.

@dmpetrov dmpetrov added this to the 0.11.0 milestone Jul 3, 2018
@erdnaavlis
Copy link

If I may step in, it would be cool to have the option for:

  1. dvc gc --stragegy LRU 10Gb - release files from cache until it is equal to, or less than 10GB.

And raise an error or a warning if it needs to release files required for current working commit/branch.

@dmpetrov
Copy link
Member Author

dmpetrov commented Jul 4, 2018

@andrethrill you are right! It is better to specify a limit for cache, then amount to release.

@ghost ghost added enhancement Enhances DVC feature request Requesting a new feature labels Feb 4, 2019
@efiop efiop added p3-nice-to-have It should be done this or next sprint and removed p4 labels Sep 25, 2019
@daavoo daavoo added A: gc Related go garbage collection and removed enhancement Enhances DVC labels Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: gc Related go garbage collection feature request Requesting a new feature p3-nice-to-have It should be done this or next sprint
Projects
None yet
Development

No branches or pull requests

4 participants