ref: clarifications around `add` and `import-url` for external targets #3210

jorgeorpinel · 2022-01-24T19:54:50Z

Related to the last task in #2121

i.e. --external, --out, and --to-remote

jorgeorpinel · 2022-01-24T20:08:47Z

content/docs/command-reference/add.md

 ## Example: Transfer to remote storage

 When you have a large dataset in an external location, you may want to track it
-as if it was in your project, but without downloading it locally (for now). The
-`--to-remote` option lets you do so, while storing a copy
-[remotely](/doc/command-reference/remote) so it can be
-[pulled](/doc/command-reference/plots) later.
+in your <abbr>project</abbr> without downloading it (yet), for example if
+there's not enough space in your current environment.


This example is almost the same as the one in the import-url ref. Since the latter has the added benefit of keeping the connection to the data source (for later updates) and to remove the docs maintenance surface, should we remove the example from this ref? I can still link to the example in import-url from the option description here.

Makes sense to me!

This example is almost the same as the one in the import-url ... should we remove the example from this ref?

Makes sense to me!

UPDATE: After the latest round of updates I'm no longer 100% sure of this since for add you need to combine --out with --to-remote. So even when the situation is very similar, the actual recipe has that key difference (there's no -o in import-url). People who never use imports may prefer to stick to add and thus need to figure out that detail.

Keeping for now!

Good point. Just to clarify, you can do dvc add --to-remote without --out:

$ dvc add --to-remote ../test.sh 100% Adding...|█████████████████████████████████████████████████|1/1 [00:00, 49.91file/s]

even when the situation is very similar, the actual recipe has that key difference (there's no -o in import-url)

I guess this wasn't that relevant after all since neither example uses -o now. The add one just notes that -o . is implied but that's not critical (already noted in the --to-remote option desc.)... We could remove it after all ⌛

content/docs/command-reference/add.md

dberenbaum · 2022-01-24T22:14:01Z

content/docs/command-reference/add.md

-The only difference that dataset is transferred straight to remote, so DVC won't
-control the remote location you gave but rather continue managing your remote
-storage where the data is now on. The operation will still be resulted with an
-`.dvc` file:
-
-```dvc
-$ ls
-data.xml.dvc
-```
-
-Whenever anyone wants to actually download the added data (for example from a
-system that can handle it), they can use `dvc pull` as usual:
+Even when nothing is downloaded locally, the operation still creates a `.dvc`
+file in the <abbr>workspace</abbr>. So whenever anyone wants to actually
+download the data, they can use `dvc pull` as usual:


Nice summarization here 🔥

per #3210 (review)

and complete console sample with ls

content/docs/command-reference/add.md

jorgeorpinel

OK @dberenbaum there's sizeable changes after your last comment so here's a summary of them 👇 Thanks

content/docs/command-reference/add.md

content/docs/command-reference/import-url.md

content/docs/command-reference/import.md

content/docs/command-reference/update.md

dberenbaum · 2022-01-29T01:26:56Z

content/docs/command-reference/add.md

-storage where the data is now on. The operation will still be resulted with an
-`.dvc` file:
+Use `--to-remote` to create a `.dvc` file for the operation without downloading
+data, transferring it directly to [remote storage] instead:


Do you think it's worth mentioning the use case here? You may not have enough storage locally, but you want to pull it later on a machine with more space.

OK but in that case it doesn't make sense to have a shorter version of the example in this ref...

Updated the motivation in both examples and reinstated everything else in the case of add. Linking to import-url for details stopped making sense (again).

dberenbaum · 2022-01-29T01:31:43Z

Thanks @jorgeorpinel! Just left a few small comments.

per #3210 (review) and #3210 (review)

and everything else while at it... Plus std. changes to corresponding import-url example

#3210) * ref: clarify external-data-related options of add i.e. --external, --out, and --to-remote * ref: improve add --out example * ref: clarify add/import-url --to-remote examples * ref: add/import-url --add option copy edits * ref: clarify add --out base case per #3210 (review) * ref: re-explain add --out example (again) * Update content/docs/command-reference/add.md Co-authored-by: Dave Berenbaum <[email protected]> * ref: rename add --out (external cache) example and and more explanation clarifications heh * ref: rewrite add/import-url --to-remote example intros to to match previous commits (improvements to add --out example text) * rephrase --to-remote examples per #3210 (review) and #3210 (review) * ref: corrections around add --to-remote and --out per #3210 (comment) * ref: std import --out description * ref: forgot to remove -o from add --to-remote example and and complete console sample with ls * ref: add ls to import-url --to-remote example * ref: consistent --to-remote examples among add and import-url * ref: more consistency changes around --to-remote * ref: remove some redundancy in import-url --to-remote example * ref: add --out/to-remote doesn't move, it copies per #3210 (review) and #3210 (review) * ref: include motivation again in add --to-remote example and and everything else while at it... Plus std. changes to corresponding import-url example Co-authored-by: Dave Berenbaum <[email protected]>

jorgeorpinel added 2 commits January 24, 2022 13:51

ref: clarify external-data-related options of add

cb290c3

i.e. --external, --out, and --to-remote

ref: improve add --out example

fff197f

jorgeorpinel changed the title ~~ref: clarifications around external-data-related operations~~ ref: clarifications around external-data-related add and import-url Jan 24, 2022

shcheklein temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 24, 2022 19:55 Inactive

jorgeorpinel changed the title ~~ref: clarifications around external-data-related add and import-url~~ ref: clarifications around add and import-url for external targets Jan 24, 2022

ref: clarify add/import-url --to-remote examples

775f8ef

jorgeorpinel marked this pull request as ready for review January 24, 2022 20:07

jorgeorpinel requested a review from dberenbaum January 24, 2022 20:07

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 24, 2022 20:07 Inactive

jorgeorpinel commented Jan 24, 2022

View reviewed changes

ref: add/import-url --add option copy edits

999e1e3

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 24, 2022 20:11 Inactive

jorgeorpinel self-assigned this Jan 24, 2022

dberenbaum reviewed Jan 24, 2022

View reviewed changes

content/docs/command-reference/add.md Show resolved Hide resolved

dberenbaum reviewed Jan 24, 2022

View reviewed changes

content/docs/command-reference/add.md Outdated Show resolved Hide resolved

dberenbaum reviewed Jan 24, 2022

View reviewed changes

content/docs/command-reference/add.md Outdated Show resolved Hide resolved

dberenbaum reviewed Jan 24, 2022

View reviewed changes

content/docs/command-reference/add.md Outdated Show resolved Hide resolved

dberenbaum reviewed Jan 24, 2022

View reviewed changes

Merge branch 'master' into ref/--to-remote

a81b9a1

jorgeorpinel added a commit that referenced this pull request Jan 25, 2022

ref: clarify add --out base case

4ac058e

per #3210 (review)

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 25, 2022 20:59 Inactive

jorgeorpinel added a commit that referenced this pull request Jan 25, 2022

ref: clarify add --out base case

3f2445b

per #3210 (review)

jorgeorpinel force-pushed the ref/--to-remote branch from 4ac058e to 3f2445b Compare January 25, 2022 21:02

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 25, 2022 21:03 Inactive

ref: clarify add --out base case

f13eed2

per #3210 (review)

jorgeorpinel force-pushed the ref/--to-remote branch from 3f2445b to f13eed2 Compare January 25, 2022 21:03

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 25, 2022 21:04 Inactive

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 25, 2022 21:21 Inactive

jorgeorpinel force-pushed the ref/--to-remote branch from a4bf375 to 1035fde Compare January 25, 2022 21:24

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 25, 2022 21:24 Inactive

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 00:19 Inactive

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 00:29 Inactive

ref: forgot to remove -o from add --to-remote example and

6e279a1

and complete console sample with ls

jorgeorpinel force-pushed the ref/--to-remote branch from af6acbb to 6e279a1 Compare January 29, 2022 00:31

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 00:31 Inactive

ref: add ls to import-url --to-remote example

77afe17

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 00:37 Inactive

ref: consistent --to-remote examples among add and import-url

1a915df

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 00:42 Inactive

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 01:08 Inactive

dberenbaum reviewed Jan 29, 2022

View reviewed changes

content/docs/command-reference/add.md Outdated Show resolved Hide resolved

dberenbaum reviewed Jan 29, 2022

View reviewed changes

content/docs/command-reference/add.md Outdated Show resolved Hide resolved

ref: more consistency changes around --to-remote

bf1cde2

jorgeorpinel force-pushed the ref/--to-remote branch from 6df4825 to bf1cde2 Compare January 29, 2022 01:15

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 01:15 Inactive

jorgeorpinel commented Jan 29, 2022

View reviewed changes

jorgeorpinel requested a review from dberenbaum January 29, 2022 01:18

ref: remove some redundancy in import-url --to-remote example

6eaba67

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 January 29, 2022 01:26 Inactive

dberenbaum reviewed Jan 29, 2022

View reviewed changes

jorgeorpinel added 3 commits February 1, 2022 16:31

Merge branch 'master' into ref/--to-remote

942f9bb

ref: add --out/to-remote doesn't move, it copies

dab0295

per #3210 (review) and #3210 (review)

ref: include motivation again in add --to-remote example and

cea4b99

and everything else while at it... Plus std. changes to corresponding import-url example

jorgeorpinel temporarily deployed to dvc-org-ref-to-remote-lyurlo62 February 1, 2022 22:58 Inactive

jorgeorpinel requested a review from dberenbaum February 1, 2022 22:58

dberenbaum approved these changes Feb 2, 2022

View reviewed changes

jorgeorpinel merged commit d5036b9 into master Feb 2, 2022

jorgeorpinel deleted the ref/--to-remote branch February 2, 2022 16:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref: clarifications around `add` and `import-url` for external targets #3210

ref: clarifications around `add` and `import-url` for external targets #3210

jorgeorpinel commented Jan 24, 2022

jorgeorpinel Jan 24, 2022

dberenbaum Jan 24, 2022

jorgeorpinel Jan 25, 2022 •

edited

Loading

dberenbaum Jan 26, 2022

This comment was marked as resolved.

jorgeorpinel Jan 29, 2022 •

edited

Loading

dberenbaum Jan 24, 2022

jorgeorpinel left a comment •

edited

Loading

dberenbaum Jan 29, 2022

jorgeorpinel Feb 1, 2022 •

edited

Loading

jorgeorpinel Feb 1, 2022

dberenbaum commented Jan 29, 2022

ref: clarifications around add and import-url for external targets #3210

ref: clarifications around add and import-url for external targets #3210

Conversation

jorgeorpinel commented Jan 24, 2022

jorgeorpinel Jan 24, 2022

Choose a reason for hiding this comment

dberenbaum Jan 24, 2022

Choose a reason for hiding this comment

jorgeorpinel Jan 25, 2022 • edited Loading

Choose a reason for hiding this comment

dberenbaum Jan 26, 2022

Choose a reason for hiding this comment

This comment was marked as resolved.

jorgeorpinel Jan 29, 2022 • edited Loading

Choose a reason for hiding this comment

dberenbaum Jan 24, 2022

Choose a reason for hiding this comment

jorgeorpinel left a comment • edited Loading

Choose a reason for hiding this comment

dberenbaum Jan 29, 2022

Choose a reason for hiding this comment

jorgeorpinel Feb 1, 2022 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Feb 1, 2022

Choose a reason for hiding this comment

dberenbaum commented Jan 29, 2022

ref: clarifications around `add` and `import-url` for external targets #3210

ref: clarifications around `add` and `import-url` for external targets #3210

jorgeorpinel Jan 25, 2022 •

edited

Loading

jorgeorpinel Jan 29, 2022 •

edited

Loading

jorgeorpinel left a comment •

edited

Loading

jorgeorpinel Feb 1, 2022 •

edited

Loading