External Data and Remotes (user-guide/external-data) #807

dashohoxha · 2019-11-20T15:19:09Z

Close #520, Close #566, Close #237, Related #108

Consolidate (and deprecate):

Maybe the examples for each remote type (those that are collapsed) should be removed as well from these man pages:

shcheklein · 2019-11-24T01:08:38Z

@dashohoxha good stuff! Will keep reviewing and asking questions.

First is - what is the difference between "Create Remotes" and "DVC storage" sections?

shcheklein · 2019-11-24T01:13:41Z

@dashohoxha I also not sure how we went from solving #520 to actually including regular remotes stuff? #520 is about very specific advanced thing - managing external data, while remotes is a very basic functionality. It deserves it's own very details section that explains them very well.

dashohoxha · 2019-11-24T04:38:50Z

what is the difference between "Create Remotes" and "DVC storage" sections

"Creates Remotes" explains all the details and options about a certain remote type (which are set with dvc remote modify).

"DVC storage" emphasizes that we need to use the option -d, --default so that the commands dvc push and dvc pull can use that remote as a "mirror" of .dvc/cache (aka a DVC Storage).

I admit that in some cases there is not much difference between them and they look very similar.

I also not sure how we went from solving #520 to actually including regular remotes stuff? #520 is about very specific advanced thing - managing external data, while remotes is a very basic functionality. It deserves it's own very details section that explains them very well.

External data and remotes seem to me closely related because:

We can use remotes to access and manage external data.
External data management is specific for each remote type. For example some remote types can be used for external outputs, some others cannot.

So, I thought that it is best to explain them together.

shcheklein · 2019-11-24T17:45:25Z

I admit that in some cases there is not much difference between them and they look very similar.

let's remove and simplify

External data and remotes seem to me closely related because:

introducing basic and advanced concepts together sends a wrong signal. We should not emphasize external deps/outs as a regular basic workflow. Also, there should be a separate motivation behind them.

dashohoxha · 2019-11-27T15:18:03Z

I admit that in some cases there is not much difference between them and they look very similar.

let's remove and simplify

I moved the sections about DVC Storages as collapsed sections under "Remotes".

introducing basic and advanced concepts together sends a wrong signal. We should not emphasize external deps/outs as a regular basic workflow. Also, there should be a separate motivation behind them.

Are you suggesting to split Remotes from External Data, like this:

user-guide/remotes/
- local
- http
- ssh
- amazon
- s3
- azure
- gs
- hdfs
- oss
user-guide/external-data/
- local
- http
- ssh
- amazon
- gs
- hdfs

shcheklein · 2019-11-27T22:34:36Z

I moved the sections about DVC Storages as collapsed sections under "Remotes".

I just don't get the purpose of those sections. Can you elaborate or explain it in some other way? Why do we need both - remotes and dvc storage?

Are you suggesting to split Remotes from External Data, like this:

Yes. I'm not sure about exact place and structure. But external data management should be separate for sure. Remotes - they can be part of the Basic Concepts, they can be part of the Managing Data section. External Data Management should be part of the Advanced section of the User Guide.

^^ this is not a precise instruction, it's just a way for me to communicate how far I think those two should be split.

dashohoxha · 2019-11-28T20:44:48Z

let's remove and simplify

I removed the sections about "DVC storage".

I also split the "Remotes" from "External Data":

The first one ("Remotes") is basically a consolidation of remote types on:

I seems like the command dvc remote only takes care of the configuration file .dvc/config (or .dvc/config.local etc.) One could modify these configuration files manually (using an editor, without using dvc remote) and everything would be fine. So, it seems more appropriate to explain different remote types on the User Guide.

The second section ("External Data") is an attempt to consolidate and improve these pages:

dashohoxha · 2019-11-28T21:01:26Z

I am not sure that I understand this note from managing-external-data:

Avoid using the same remote location that you are using for dvc push, dvc pull, dvc fetch as external cache for your external outputs, because it may cause possible checksum overlaps. Checksum for some data file on an external storage can potentially collide with checksum generated locally for a different file, with a different content.

The first part seems reasonable: "Avoid using the same remote location".

The explanation seems nonsense. If there is some possibility for checksum collision, this may happen for the cached files as well (two different cached files happen to have the same checksum). If those external data files that we are tracking with DVC were instead local files, we would store them in cache with all the other project files. So, what difference does it make if they are tracked remotely?

Maybe some other explanation is needed or maybe we should remove this note.

shcheklein · 2019-11-28T21:13:08Z

src/Documentation/sidebar.json

+        ]
+      },
+      {
+        "label": "External Data",


why can't we keep the Managing External Data label and name? Not much difference but redirect won't be required.

why can't we keep the Managing External Data label and name? Not much difference but redirect won't be required.

Redirect would still be required because of "external-dependencies".
On the other hand "External Data" is more generic than "Managing External Data". For example in the case of HTTP (and other cases) we are just accessing or downloading external data, "managing" doesn't seem suitable.

Again, please, don't resolve. I have to repeat myself and create duplicates.

Don't see any reasons to rename and complicate this.

Don't see any reasons to rename and complicate this.

I don't understand, what is the complication that comes from renaming? In the first comment you say "not much difference", and I agree with this. And redirect is still required even if we don't change the name.

one extra redirect to support + updating links in the docs

I have already updated the links on the docs, and also have fixed the redirect.
Changing the name back would be extra work, and unjustified, unless the old name is better than the new one.

shcheklein · 2019-11-28T21:14:17Z

src/Documentation/sidebar.json

@@ -120,6 +120,80 @@
        "label": "Managing External Data",
        "slug": "managing-external-data"
      },
+      {
+        "label": "Remotes",


Come up with a better name - Setting up storage?

Move the section up.

Come up with a better name - Setting up storage?

I am renaming to "DVC Remotes". I am using "DVC Storage" for the mirror of .dvc/cache/. A "DVC Remote" is analogous to a Git Remote, and is broader than a "DVC Storage" (which is just a mirror/backup of the local cache). This terminology is just a convention (for example instead of "DVC Storage" we could have used "DVC Cache Backup".

Please don't resolve comments that are not created by you. As I mentioned already, it's extremely hard to follow and track changes this way.

I would prefer to have something human readable . Otherwise we are defining remote through remotes. Remote is remote ... kind-a of thing.

I would prefer to have something human readable . Otherwise we are defining remote through remotes. Remote is remote ... kind-a of thing.

I don't understand this. We are talking about the remotes that are managed with the command dvc remote. What could be a better name than "Remote" or "DVC Remote"?

src/Documentation/sidebar.json

shcheklein · 2019-11-28T21:16:41Z

src/Documentation/sidebar.json

+        "source": "external-data/index.md",
+        "children": [
+          {
+            "label": "Local Filesystem",


Local should go very last in all cases. SSH, S3, Google Cloud, Azure, HTTP, HDFS - something like this is better

Local supports more cases than the other types, and it also can be used for the mounted data storages (NAS etc.) So, it seems like an important one. Why should it go last?

static/docs/user-guide/external-data/amazon.md

shcheklein · 2019-11-28T21:19:10Z

static/docs/user-guide/external-data/amazon.md

+$ dvc import-url s3://mybucket/file.csv
+```
+
+Or, using a remote:


remote -> remote notation?

or put remote

it's kinda not only remote as usual it's also a protocol, so some explanation is needed

remote -> remote notation?

First we create a remote (with dvc remote add), so "remote" is ok.
I think that there is no need to introduce the term "remote notation", because it is kind of obvious.

shcheklein · 2019-11-28T21:20:18Z

static/docs/user-guide/external-data/amazon.md

+
+## External Data and Outputs
+
+For cached external outputs (specified using `-o`) we need to setup an external


dvc add creates an output as well

static/docs/user-guide/external-data/gs.md

shcheklein · 2019-11-28T21:22:44Z

static/docs/user-guide/external-data/hdfs.md

@@ -0,0 +1,89 @@
+# External Data on HDFS


let's remove all external data section for HDFS. We don't support it actually and it should be remove from DVC for now. cc @efiop can confirm that.

Confirmed, we were not able to find a suitable workaround for external outputs. 🙁 So let's hide hdfs external deps and outs from docs for now. We'll probably remove them from dvc too very soon.

So let's hide hdfs external deps and outs from docs for now.

I am removing the section "External Data and Outputs". But it seems like "External Dependencies" should work. Please correct me if I am wrong.

@dashohoxha even external dependencies do not work reliable unfortunately as far as I remember

shcheklein · 2019-11-28T21:24:09Z

static/docs/user-guide/external-data/http.md

@@ -0,0 +1,66 @@
+# External Data on HTTP/HTTPS
+
+**❗ Note:** HTTP/HTTPS remotes support only download operations.


it means probably that we support only dependencies, right? use the same terminology ... not download operations

it means probably that we support only dependencies, right? use the same terminology ... not download operations

I think it is better to remove this note, since it says the obvious (there is only the section for dependencies, no section for outputs).

We've seen people asking us over and over again - thus this note. It was not cleat either docs are out of date (always) or what :)

Maybe it was useful when HTTP was in the same page with the other remotes, but in this page it does not make sense.
Probably people are confused by Git remotes, which can be used both to pull and push.

shcheklein · 2019-11-28T21:25:26Z

static/docs/user-guide/external-data/index.md

@@ -0,0 +1,22 @@
+# External Data
+
+External data are located outside the project directory, either on the local


I think we had better explanation before why and when it should be used.

Do you mean this explanation:

There are cases when data is so large, or its processing is organized in a way that you would like to avoid moving it out of its external/remote location. For example from a network attached storage (NAS) drive, processing data on HDFS, running Dask via SSH, or having a script that streams data from S3 to process it. A mechanism for external outputs and external dependencies provides a way for DVC to control data externally.

shcheklein · 2019-11-28T21:25:46Z

static/docs/user-guide/external-data/index.md

+# External Data
+
+External data are located outside the project directory, either on the local
+filesystem or on a remote host. They may be used in order to:


remote host sounds like a machine while it can be actually a cloud

remote host sounds like a machine while it can be actually a cloud

"remote host" (like a machine) is one the cases. I am also adding "cloud data storage", like this:

External data are located outside the project directory, either on the local
filesystem, or on a remote host, or on a cloud data storage.

shcheklein · 2019-12-14T17:12:23Z

Again, too complicated, too many changes at once, no luck of iterating on this not willingness to address even small changes.

shcheklein temporarily deployed to dvc-org-pr-807 November 20, 2019 15:19 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 22, 2019 05:26 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 22, 2019 05:49 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 22, 2019 14:56 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 22, 2019 21:36 Inactive

dashohoxha changed the title ~~[WIP] External Data and Remotes (user-guide/external-data)~~ External Data and Remotes (user-guide/external-data) Nov 22, 2019

weekly-digest bot mentioned this pull request Nov 24, 2019

Weekly Digest (17 November, 2019 - 24 November, 2019) #815

Closed

shcheklein temporarily deployed to dvc-org-pr-807 November 24, 2019 07:43 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 27, 2019 15:11 Inactive

shcheklein temporarily deployed to dvc-org-pr-807 November 28, 2019 20:29 Inactive

shcheklein reviewed Nov 28, 2019

View reviewed changes

src/Documentation/sidebar.json Outdated Show resolved Hide resolved

shcheklein reviewed Nov 28, 2019

View reviewed changes

static/docs/user-guide/external-data/amazon.md Outdated Show resolved Hide resolved

shcheklein reviewed Nov 28, 2019

View reviewed changes

static/docs/user-guide/external-data/gs.md Outdated Show resolved Hide resolved

shcheklein reviewed Nov 28, 2019

View reviewed changes

dashohoxha added 16 commits December 1, 2019 00:57

Details for HTTP

176c8e6

Make remote subheaders uniform

99b17e3

Add import-url examples to external dependencies

6c388c1

Reorganize the sections of external data and outputs

64b055b

Replace "S3 API" with "S3 Compatible"

84d2a42

Rename "S3 API" to "S3 Compatible"

f68351b

Move the section of "DVC Storage" as a collapsed section under "Remotes"

10ce9d8

Split remotes and external-data in separate sections

b92192b

Remove remote sections about DVC storage

caa9dbb

Remove old sections from the sidebar

06208c0

Redirect old pages to the new ones

39e17a2

Remove warning notes and descriptions of the storage services

a1cc516

Some corrections and improvements

eb1082a

HDFS cannot be used for external data, remove its page

b4801e9

Unhide the sections of remote options

6ed975b

Remove remote details from the command reference

2f2f9cf

dashohoxha force-pushed the external-data branch from 8cf733d to 2f2f9cf Compare December 1, 2019 00:10

shcheklein temporarily deployed to dvc-org-pr-807 December 1, 2019 00:10 Inactive

Merge branch 'master' into external-data

4e6c3ef

shcheklein temporarily deployed to dvc-org-pr-807 December 1, 2019 00:13 Inactive

Move the section of Local DVC Storage to remotes

1dcd331

shcheklein temporarily deployed to dvc-org-pr-807 December 1, 2019 01:05 Inactive

dashohoxha requested a review from shcheklein December 1, 2019 01:27

weekly-digest bot mentioned this pull request Dec 8, 2019

Weekly Digest (1 December, 2019 - 8 December, 2019) #842

Closed

shcheklein temporarily deployed to dvc-org-pr-807 December 9, 2019 17:03 Inactive

shcheklein closed this Dec 14, 2019

efiop deleted the external-data branch December 15, 2019 14:21

efiop restored the external-data branch December 15, 2019 14:21

jorgeorpinel deleted the external-data branch January 23, 2020 17:01


		## External Data and Outputs

		For cached external outputs (specified using `-o`) we need to setup an external

		@@ -0,0 +1,66 @@
		# External Data on HTTP/HTTPS

		❗ Note: HTTP/HTTPS remotes support only download operations.

		@@ -0,0 +1,22 @@
		# External Data

		External data are located outside the project directory, either on the local

External Data and Remotes (user-guide/external-data) #807

External Data and Remotes (user-guide/external-data) #807

Conversation

dashohoxha commented Nov 20, 2019 • edited Loading

shcheklein commented Nov 24, 2019

shcheklein commented Nov 24, 2019

dashohoxha commented Nov 24, 2019

shcheklein commented Nov 24, 2019

dashohoxha commented Nov 27, 2019

shcheklein commented Nov 27, 2019

dashohoxha commented Nov 28, 2019

dashohoxha commented Nov 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shcheklein commented Dec 14, 2019

dashohoxha commented Nov 20, 2019 •

edited

Loading