Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ML Pipelines: recommend push=false for intermediate outputs #4267

Closed
Tracked by #2883
jorgeorpinel opened this issue Jan 26, 2023 · 1 comment
Closed
Tracked by #2883

ML Pipelines: recommend push=false for intermediate outputs #4267

jorgeorpinel opened this issue Jan 26, 2023 · 1 comment
Labels
A: docs Area: user documentation (gatsby-theme-iterative) C: guide Content of /doc/user-guide C: start Content of /doc/start p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately.

Comments

@jorgeorpinel
Copy link
Contributor

Echoing iterative/dvc#8581 (comment):

Should we plan to recommend this a lot in Data Pipeline docs? Specifically for intermediate pipeline outputs. Assuming the happy path out there is to push only raw data and likely final ML model files (everything else may be best to dvc repro when needed).

If we don't at least emphasize the possibility, users may realize too late they have pushed a bunch of intermediate output versions and they are pretty difficult to clean up with dvc gc (support example).

Originally posted by @jorgeorpinel in #4142 (comment)

See push field in https://dvc.org/doc/user-guide/project-structure/dvcyaml-files#output-subfields and https://dvc.org/doc/user-guide/project-structure/dvc-files#output-entries.

@jorgeorpinel jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) C: guide Content of /doc/user-guide labels Jan 26, 2023
@jorgeorpinel jorgeorpinel changed the title guide: recommend push=false for intermediate outputs ML Pipelines: recommend push=false for intermediate outputs Jan 26, 2023
@jorgeorpinel jorgeorpinel added the C: start Content of /doc/start label Jan 26, 2023
@jorgeorpinel
Copy link
Contributor Author

let's brainstorm more on it (#4142 (comment))

Cc @dberenbaum hoping this issue is helpful for that purpose. Added task in #2883 for now too.

@dberenbaum dberenbaum added the p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately. label Oct 16, 2023
@dberenbaum dberenbaum closed this as not planned Won't fix, can't repro, duplicate, stale Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: docs Area: user documentation (gatsby-theme-iterative) C: guide Content of /doc/user-guide C: start Content of /doc/start p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately.
Projects
None yet
Development

No branches or pull requests

2 participants