Model registry docs

[dberenbaum]

Report

Update the model registry docs to combine all the products (GTO, DVC, Studio) into a coherent workflow and lead DVC users to this workflow. Some possible places to update:

• Explain artifacts section in https://dvc.org/doc/user-guide/project-structure/dvcyaml-files
• Explain in https://dvc.org/doc/dvclive/api-reference/live/log_artifact
• Explain in https://dvc.org/doc/dvclive/how-it-works#track-large-artifacts-with-dvc
• https://mlem.ai/doc/gto/ (see WIP: Docs draft for integration with DVC mlem.ai#323)
• https://dvc.org/doc/studio/user-guide/model-registry/
• https://dvc.org/doc/studio/get-started#manage-models
• Mention in https://dvc.org/doc/start/experiments/experiment-versioning
• Mention in https://dvc.org/doc/user-guide/experiment-management
• Links in https://dvc.org/doc/use-cases/model-registry
• Update example repos Register models in the example repos to see them in the MR example-repos-dev#203
• Update get/import in https://dvc.org/doc/command-reference/get and https://dvc.org/doc/command-reference/import
• as discussed in Add API to add artifacts dvc#9345, providing good CI/CD examples is a priority
• Add log_artifact examples for at least the most popular frameworks in https://dvc.org/doc/dvclive/ml-frameworks

Thoughts @aguschin @tapadipti @shcheklein?

[shcheklein]

Yep, I like it. Two things come to my mind:

• It should be less about GTO and MLEM. Ideally we don't even mention them. Or only as a supportive tools that solve a very specific task inside, etc. It's not because I don't like them, etc. It's because I feel it's too complicated and confusing for users too have all of them at once in all those pages.
• I would prioritize some strong user story in the Studio docs - DVC and GTO both - are implementation details in this case.
• DVC Use case can include reference to Studio as well

[dberenbaum]

@aguschin Let's prioritize updating the DVC docs first so we have somewhere to point people with the "correct" usage today.

[dberenbaum]

We have a couple people asking about/waiting on docs in discord: https://discord.com/channels/485586884165107732/1096339522553384990/1096457228258652370

[tapadipti]

Sounds good @dberenbaum. In the Studio docs, we can restructure some of the content to unhighlight GTO, MLEM.

I would prioritize some strong user story in the Studio docs

@shcheklein Any suggestions on what user story to follow?

The current structure does follow user workflows: add models, register versions, assign stages, download files, and once we have the functionality - deploy models. Any suggestions on a better flow?

[shcheklein]

The missing part that can be useful - find the model with the right metrics before promoting / compare models.

[dberenbaum]

Need to consider where to keep GTO docs

[dberenbaum]

Added an item for https://dvc.org/doc/studio/get-started#manage-models

[dberenbaum]

Updated to reflect #4481. @aguschin Please update anything I missed.

[dberenbaum]

Thanks to @aguschin for moving this forward! After taking a look at #4516, I have a better idea of what else might be useful here:

• https://dvc.org/doc/use-cases/model-registry
  • focus on the end-to-end workflow (add/log->compare->register->assign/promote->download->deploy)
  • focus primarily on studio and maybe mention dvclive as starting point, less focus on other individual tools (gto, mlem, dvc)
• https://dvc.org/doc/studio/user-guide/model-registry/what-is-a-model-registry - same as above
  • focus on the end-to-end workflow (we need a full end-to-end tutorial somewhere, we can start with a video)
  • less focus on GTO/MLEM (although fine to mention/link to them when they are useful, like as an example of how to deploy)
• https://dvc-org-dvc-artifacts-8epucpc9.herokuapp.com/doc/studio/user-guide/model-registry/add-a-model
  • add separate sections for adding from GUI/Python/CLI
  • simplify instructions -- good to have detail should be able to show a simple basic workflow in fewer steps
  • update video
• https://dvc.org/doc/studio/user-guide/model-registry/view-models
  • show how to compare models
• https://dvc.org/doc/studio/user-guide/model-registry/download-models
  • Show how to get path and/or version with and without studio
  • Mention GTO action and how to use it
  • Show how to use this in automation/deployment (mention MLEM)
  • Move below register and assign pages (more natural workflow)
• https://dvc.org/doc/studio/user-guide/model-registry/register-version
  • Like add a model, details are good but basic workflow looks too complex
• https://dvc.org/doc/studio/user-guide/model-registry/assign-stage
  • Same as above (basic workflow looks complex)
• "we are in the process of building MR<" -> we built it
• add log_artifact to the examples in https://dvc.org/doc/dvclive/ml-frameworks.

[dberenbaum]

• Add log_artifact examples for at least the most popular frameworks in https://dvc.org/doc/dvclive/ml-frameworks

Added this item to the list @aguschin

[aguschin]

Ok, another PR that updates some parts of this #4587

Except for that, what's left:

• Add log_artifact examples for at least the most popular frameworks (dvclive) - somewhat depends on what we decided in Decouple dvc add from log_artifact dvclive#572
• update example repos Register models in the example repos to see them in the MR example-repos-dev#203

two moments to discuss - wdyt @dberenbaum?

• Not sure if this page needs a note about adding models to dvc.yaml https://dvc.org/doc/start/data-management/metrics-parameters-plots
• "add separate sections for adding from GUI/Python/CLI" to https://dvc.org/doc/studio/user-guide/model-registry/add-a-model - not sure how to do that without copy-pasting things from other parts of docs. For now I consider that it's enough to have a sentence and links to right places in docs. Can try to add 1-2 sentences to elaborate (just for the sake of first-time users visiting this page may not know about DVClive/DVC and what it does).

[dberenbaum]

Thanks @aguschin! I think what you covered is a good start. Once you address the things you mention, I'm fine to close this and follow up with smaller issues for anything else.

• Add log_artifact examples for at least the most popular frameworks (dvclive) - somewhat depends on what we decided in Decouple dvc add from log_artifact dvclive#572

I don't think it's a release blocker, but I'd suggest spending time trying it in more frameworks will help unblock iterative/dvclive#572. I'd also consider prioritizing iterative/dvclive#586, since having a model-specific method might also help unblock things.

• Not sure if this page needs a note about adding models to dvc.yaml https://dvc.org/doc/start/data-management/metrics-parameters-plots

I would leave it out. That page unfortunately needs a lot of work, and I don't think adding another concept to it will help.

[dberenbaum]

@aguschin Should we consider this closed and open follow-ups for anything remaining? Do you see any blockers for release?

[aguschin]

No, we don't have blockers here I think.
Created a follow-up #4622
Thanks for the help on this @dberenbaum!