log_artifact

[dberenbaum]

Similar to https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.log_artifact, dvclive can have a log_artifact method, which can do dvc add to track an artifact.

[shcheklein]

@tapadipti @dmpetrov @aguschin I mentioned this in the conversation on the model registry. I think this workflow should be supported. It's a way for people to dump their models to Studio, or to DVC first and connect in Studio, etc. I think this is the basic approach that MLFlow, W&B, etc are taking and it makes sense to me:

• it's aligned with DVCLive efforts and Live metrics, etc.
• It's aligned with simplifying MR experience - we can show a single line people would need to copy into their Notebook to start
• it's even aligned with an effort of disconnecting it from Git to the same extent as Live experiments

Related - #305 (but I would not be making this about MLEM - I think it's an optional advanced feature for now there). log_model can exist by itself.

[daavoo]

do dvc add to track an artifact.

🤔 top-level outputs so that they can be added to the dvc.yaml of #366 ?

[tapadipti]

So a dependency for this would be that the user has dvc set up, right? Should it also be possible to log_artifact without dvc tracking (like log_metric)?

For registering new versions of an existing artifact, we'll need an option to provide an existing artifact identifier (and possibly a new version number).

log_model can exist by itself

It could also be a type=model option to log_artifact, right? We will need the artifact type for the model registry anyway.

@shcheklein some thoughts about this simplifying the MR experience:

It's aligned with simplifying MR experience - we can show a single line people would need to copy into their Notebook to start

The users would have to leave the Studio UI and edit their ML code/training to start registering models. And they cannot register models after they have been created. Currently, there's dependency on GTO; with this there will be dependency on DVCLive. Also, they'll still need to add the Git repo in Studio (assuming we don't provide a Studio server endpoint to log artifacts). So, it'll probably not be simple.

it's even aligned with an effort of disconnecting it from Git to the same extent as Live experiments

Are you suggesting that we will provide a Studio server endpoint for all users to log their artifacts to? And Studio will maintain a database of all artifacts instead of users saving it to their Git repos? (This is how the live metrics are saved).

[shcheklein]

Are you suggesting that we will provide a Studio server endpoint for all users to log their artifacts to? And Studio will maintain a database of all artifacts instead of users saving it to their Git repos? (This is how the live metrics are saved).

Yes, it should be an available alternative for this.

[dberenbaum]

@shcheklein @tapadipti I was thinking the starting point here would be to still rely on DVC and Git here and save it locally by starting to track the artifact with DVC.

Some follow-up steps could be:

1. Providing some Studio storage so a user could dvc push the artifact without setting up their own cloud remote.
2. Auto-push the artifact along with metrics, parameters, and plots. Does it make sense to use dvc exp push functionality at that point, or do we want two different ways to push experiment results?

What do you think?

🤔 top-level outputs so that they can be added to the dvc.yaml of #366 ?

@daavoo I'm fine to start with saving in .dvc files, especially since the artifacts themselves aren't likely to be inside the dvclive directory and it should be easier than adding a new dvc.yaml section and dvclive-based dvc.lock.

[dberenbaum]

Lowering to p2 since I think we still need to better scope what's needed here, and it doesn't break any current workflows

[daavoo]

Lowering to p2 since I think we still need to better scope what's needed here, and it doesn't break any current workflows

The save_dvc_exp feels broken/incomplete to me without something like log_artifact (even just with a very limited scope based on dvc add + ensure .dvc is in include_untracked), mostly because of the scenario described in iterative/example-repos-dev#171

[tapadipti]

This will also be needed at some point to simplify the model registry onboarding experience (although we've not scoped it out yet).

[dberenbaum]

even just with a very limited scope based on dvc add + ensure .dvc is in include_untracked

Okay, let's add it back if it's needed there with the limited scope you mention for now.

I'd also like to address iterative/dvc#8986 if we do this to improve the transition to pipelines.