Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log_artifact: add cache option, only write to dvc.yaml if metadata ex… #620

Merged
merged 1 commit into from
Jul 10, 2023

Conversation

dberenbaum
Copy link
Collaborator

@dberenbaum dberenbaum commented Jul 5, 2023

Extracted from #613.

  • Artifacts will only be annotated in dvc.yaml:artifacts if some metadata is provided (type, name, desc, labels, or meta). This is a breaking change, but I can't see how anyone is making use of this without any metadata since it won't be used by the model registry. It's needed in auto save model in lightning #613 so we can cache a whole directory of checkpoints without annotating the whole directory.
  • Added cache kwarg to log_artifact (defaults to True) so that it's possible to annotate the artifact metadata without caching the object. It's needed in auto save model in lightning #613 so we can annotate the best artifact in dvc.yaml without re-caching (since it overlaps with the cached checkpoints directory).

There are probably workarounds for both in #613, but I think this flexibility to choose when to cache and when to annotate in dvc.yaml is useful. It also should close #572 and #551 (edit: and partially address #551) since it decouples caching and annotation.

@dberenbaum
Copy link
Collaborator Author

Looks like there are unrelated errors coming from the keras callback.

Copy link
Contributor

@daavoo daavoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree it closes #572
But I don't think it closes the external part of #551

@daavoo daavoo force-pushed the log-artifact-updates branch from 1b4549f to 57e273b Compare July 6, 2023 15:24
@dberenbaum dberenbaum merged commit dfff9c4 into main Jul 10, 2023
@dberenbaum dberenbaum deleted the log-artifact-updates branch July 10, 2023 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Decouple dvc add from log_artifact
2 participants