guide: Data Management

[iesahin]

UPDATE: #2856 (comment)

---

This is the plan for data management trail that focuses on:

• Adding data to DVC projects
  & Versioning data in DVC projects
• The cache (local)
• & Shared cache (external)
• Removing data from DVC projects
• + Creating remotes (link to config/remotes)
• & Sync with remotes
  See also guide: extract remote add/modify details from cmd ref. #2866
• Accessing public datasets and data registries (get, import)
• External data topics (See how: use DVC when data is stored in an external drive #563 (comment))
  • probably address guide: consolidate external data mgmt guides #520

Adding data to DVC projects

• Initialize a DVC repository and use dvc add to add files.

• We'll assume MNIST data exist in a folder and will add it.

Versioning data in DVC projects

• Overwrite Fashion-MNIST data on top of MNIST and update the dataset.
• Go back and forth in Git history to get different datasets in the same folder.

Creating remotes

• Add a Google Drive folder as a remote.

• Make it default

Pushing to/pulling from remotes

• Push the cache to the remote we created
• Clone the repository to somewhere (e.g. ssh or local folder)
• Pull the cache

Accessing public datasets and registries

• Get the Fashion MNIST data from dataset-registry

Removing data from DVC projects

• Remove certain folders from workspace
• Delete the corresponding cache files

UPDATE: start with a reorg, see #2856 (comment) below (may be enough).

[jorgeorpinel]

re title I'd just avoid the phrase "model management" since it has a specific meaning (not what this is about) but if you want to include "model" maybe use "data and model file management". I don't think we need to include "model" in the title but "model file(s)" can be included in the content.

[shcheklein]

since it has a specific meaning

I'm not sure we have it written somewhere? :) what kind of meaning do you have? what is so different between model management and data management?

(I can see for example that if we include model management we could keep some parts about metrics for example - which is also fine)

[jorgeorpinel]

Like, related to the ML model lifecycle? I mentioned this in the PR (https://www.dominodatalab.com/solutions/model-management/) and it wasn't contested so I assumed I was correct but you guys are the experts! If it doesn't have a special meaning then it doesn't matter. But if it does users and search engines could get confused.

[iesahin]

I'm not an expert on naming things :) I put "model" to the title because the current docs have it, and we put "model" after a user requested it. I understand the meaning described in https://www.dominodatalab.com/solutions/model-management/ and how it differs from the way we use it, but in this new domain usually people use the same words to mean different things.

I have no strong opinion here, and honestly writing a specific "model management" document to the UG might be more appropriate. But until then, we can have "model" in the title and we can say that "models are files that can be tracked by DVC" in the text.

[jorgeorpinel]

Data Mgmt is simple enough that covering it in the Get Started and Command Reference has thus far been enough. But having a group of existing content under this "Category" could achieve some goals:

• Help reorg existing content (e.g. Large Dataset Optimization, Managing External Data, etc.)

• Include Remote usage details? See guide: extract remote add/modify details from cmd ref. #2866
  Or should that be part of a Configuration guide? (rel. guide: extract part of theconfig cmd ref. #340)

So just doing that reorg of existing content could be a good and quick first step, I think. Then we reconsider all the material proposed above. WDYT?

[jorgeorpinel]

re Help reorg existing content

Based on the OP here's what we currently have for all the topics mentioned:

Adding data to DVC projects
& Versioning data in DVC projects

• Covered in https://dvc.org/doc/use-cases/versioning-data-and-models
• and intro-ed in https://dvc.org/doc/start/data-management/data-versioning

The cache (local)

• https://dvc.org/doc/user-guide/project-structure/internal-files and
• https://dvc.org/doc/user-guide/data-management/large-dataset-optimization#configuring-dvc-cache-file-link-type
  + https://dvc.org/doc/command-reference/config#cache

& Shared cache (external)

• https://dvc.org/doc/user-guide/how-to/share-a-dvc-cache
  + https://dvc.org/doc/command-reference/gc#cleaning-shared-cache-or-remote
• https://dvc.org/doc/user-guide/data-management/managing-external-data#setting-up-an-external-cache

Removing data from DVC projects

• https://dvc.org/doc/user-guide/how-to/stop-tracking-data
  + https://dvc.org/doc/command-reference/remove, https://dvc.org/doc/command-reference/gc

Creating remotes (link to config/remotes)
& Sync with remotes

• Intro-ed in https://dvc.org/doc/start/data-management/data-versioning#storing-and-sharing
• More info. in refs under https://dvc.org/doc/command-reference/remote

Accessing public datasets and data registries (get, import)

• Intro-ed in https://dvc.org/doc/start/data-management/data-and-model-access
• Also covered in https://dvc.org/doc/use-cases/data-registry

External data topics

• https://dvc.org/doc/user-guide/data-management/importing-external-data
• https://dvc.org/doc/user-guide/data-management/managing-external-data
  + in refs e.g. https://dvc.org/doc/command-reference/add#example-transfer-to-remote-storage

[jorgeorpinel]

Given that all that is already covered (albeit maybe disorganized) and not really the general goal of the UG (explanation-type docs), here's a new plan for the Data Management user guide:

• Motivation: Why do you need to manage data better (using DVC and it's Git-based approach)
  Maybe use a diagram (related to this one).
  guide: Data Management #4042

• Form a perspective: Give clear overview of the pieces involved: workspace, Git, cache, remote storage, + basic and sync operations (add,remove,push,pull,etc) ?
  guide: Basic Operations (Data Mgmt) #4053

• Secondary: Maybe concentrate info on some of these parts and features from existing docs (check refs, how-tos, etc.)

• Idea: Emphasize the value by contrasting typical/ ad hoc methods vs DVC project structures (before/after)

  Currently people may struggle to understand the difference; Note: don't overcomplicate this though (shouldn't look scary).

• Explain remote storage properly (scenarios, setup, sync, cleanup, etc.)
  Inc. guide: extract remote add/modify details from cmd ref. #2866
  guide: remote storage #4058

• Other applications i.e. shared cache, external data, etc.
  Inc. guide: consolidate external data mgmt guides #520

[jorgeorpinel]

Idea: Emphasize the value by contrasting typical/ ad hoc methods vs DVC project structures (before/after)

My only problem with this idea is that we should drive the value of the product and feature earlier than the user guide. This should be in use cases or in the Get Started if needed, even in README as well. OK to repeat in the UG as well, but as a quick recap. @shcheklein