guide: consolidate external data mgmt guides

[shcheklein]

done

• External dependencies and outputs.

next

• Add proper links through out other doc sections.
• Explain external cache here or in the cache docs but not in the shared dev server use case, because shared cache isn't compatible with external data (see support adding/transfering data straight to cache/remote  dvc#4520 (comment))
• Clarify about cache link types per remote location as discussed in mention that cache options could be used besides remote modify params #764.
• Decide whether how-to is also needed per how: use DVC when data is stored in an external drive #563 (comment)

later?

• Move them under single Manage External Data section (rel docs: "definitive" organization #144) with some motivation in the index. [or rewrite completely?]
• Clarify differences and use cases for external storage mechanisms (see user-guide: clarify differences and use cases for external storage mechanisms? #566) [terminology]
• Add info about NFS (see guide: using NFS as a remote storage #103) [remote-related]
• Probably also address guide: explain about encrypted buckets in external data mgmt #774 about encrypted buckets

[dashohoxha]

I think this structure looks too complicated and is not correct.

I agree. After thinking a bit about it, it seems that that structure might be suitable for a page/section about "Remote Storage Types", like this:

• Remote Storage Types
  • Local Files and Directories
  • SSH
  • Amazon S3
  • Google Cloud Storage
  • HDFS
  • HTTP

Most important here is to explain when and why the external data management is needed.

Exactly. For example I am not sure when a remote cache might be useful. I can think of two cases:

1. When the remote cache is actually a local directory.
2. When the project is used just for dataset management (there are no stages).

Have I missed any other cases when it might be a preferred choice? The case that is used currently as an example (copying datafiles to a remote storage) does not seem convincing.

[shcheklein]

I think that Remote Storage Types by itself can and should be part of the "Data Sharing" section you made.

For example I am not sure when a remote cache might be useful.

In general you need all "external stuff" when you have some processing system that writes/read directly into/from it. E.g Spark that runs on top of S3 and crunches data that is too large to bring it to local machine. It means some mixed cases - e.g. raw data is large and you read it directly from S3 but intermediate artifacts, models, etc are stored locally.

So, rule of thumb - think about script in any language. If it internally has a pointer to S3, or any other cloud/remote storage (relative to the location you run this script from) it means you need some external data management.

[dashohoxha]

I have a question.

If you check the new section External Data and Remotes, you will notice that some remote types (SSH, Amazon S3, GS, HDFS) have subsections about external dependencies and outputs, while the others (S3 API, Azure, Aliyun OSS) do not have such subsections.

This is simply because their information is missing from the pages that are being consolidated:

• https://dvc.org/doc/user-guide/external-dependencies
• https://dvc.org/doc/user-guide/managing-external-data

I suspect that external dependencies and outputs work the same way for these remote types as well. However I cannot test this for each remote type (I am just reorganizing and rewriting the information that is already available on these pages and on the man pages of dvc remote).

Can someone confirm that the external dependencies and outputs work the same way for all the remote types (in particular for S3 API, Azure and Aliyun OSS)?

[shcheklein]

@efiop has more up to date info on this in terms of specific remotes supporting specific features. But in general, that information is not present because on certain types of remotes we do not support external deps/outs yet. May be even some table would be beneficial with a summary of features supported by different remotes.

S3 API name is a bit confusing btw. It should be something like - S3 Compatible or may be there is a specific term for that.

[jorgeorpinel]

Added a checkbox about the external cache section (per #654 (comment))

[dberenbaum]

Not sure where is best to add this, but we continue to see confusion about how to use DVC with externally stored data, especially when working in cloud-based notebook environments like Databricks, Sagemaker, and Colab. See the discussions below:

• DBFS (Databricks) Support dvc#4336 (comment)
• https://discord.com/channels/485586884165107732/563406153334128681/1088652943584153673
• https://discord.com/channels/485586884165107732/1089910570888724520/1089933356269047869

[dberenbaum]

Another report of this confusion: https://discord.com/channels/485586884165107732/485596304961962003/1095518763228610570.

Making this p1. I will try to reword/reorganize the info here.