ref: document `import-url` cloud versioning changes #4142

pmrowla · 2022-11-29T10:29:11Z

Per

github-actions · 2022-11-29T10:40:45Z

Link Check Report

There were no links to check!

dberenbaum · 2022-11-29T15:24:49Z

content/docs/command-reference/import-url.md

+  file. By default, DVC will automatically capture cloud versioning information
+  if the URL contains a cloud versioning ID. When `--version-aware` is provided
+  along with a URL that does not contain a cloud versioning ID, DVC will capture
+  the latest version of the file.


Let's also explain that dvc will pull that version from the source location even if it's overwritten, and will not push another copy of it to the remote.

cc @jorgeorpinel Is there somewhere in the data management user guide we want to this info also?

We'll def. need UG updates to go over cloud versioning (feel free to make a separate docs issue) -- can't explain everything in an option text. For now I'd focus on what the flag does, and put some explanations in the Description (which in this case is already super long and should be rewritten/ moved to UG eventually).

p.s. Specifically, there's is some draft content about this in https://github.com/iterative/dvc.org/pull/4119/files#diff-d01612907e4ab14238625d537eaf42852d8566901d1bfe2fd3f2d4406a2d1dfc ATM.

jorgeorpinel

Committing...

content/docs/user-guide/project-structure/dvc-files.md

content/docs/user-guide/project-structure/dvcyaml-files.md

Co-authored-by: Restyled.io <[email protected]>

jorgeorpinel · 2022-12-03T23:23:45Z

content/docs/user-guide/project-structure/dvcyaml-files.md

@@ -265,6 +265,7 @@ These include a subset of the fields in `.dvc` file
 | `persist`    | Whether the output file/dir should remain in place during `dvc repro` (`false` by default: outputs are deleted when `dvc repro` starts)                                                                                                                                           |
 | `checkpoint` | (Optional) Set to `true` to let DVC know that this output is associated with [checkpoint experiments](/doc/user-guide/experiment-management/checkpoints). These outputs are reverted to their last cached version at `dvc exp run` and also `persist` during the stage execution. |
 | `desc`       | (Optional) User description for this output. This doesn't affect any DVC operations.                                                                                                                                                                                              |
+| `push`       | Whether or not this file or directory, when previously <abbr>cached</abbr>, is uploaded to remote storage by `dvc push` (`true` by default).                                                                                                                                      |


Echoing iterative/dvc#8581 (comment):

Should we plan to recommend this a lot in Data Pipeline docs? Specifically for intermediate pipeline outputs. Assuming the happy path out there is to push only raw data and likely final ML model files (everything else may be best to dvc repro when needed).

If we don't at least emphasize the possibility, users may realize too late they have pushed a bunch of intermediate output versions and they are pretty difficult to clean up with dvc gc (support example).

I'm not sure that not pushing is the right default behavior, even for intermediate outputs. If the user wants to take advantage of run-cache to not re-run stages that have already been reproduced, they still need to push/pull intermediate outs

@jorgeorpinel Thinking about it some more, I like the suggestion and think it makes sense as a possible product direction to make it easier to get started with pipelines, so let's brainstorm more on it.

dberenbaum · 2022-12-05T13:21:14Z

Related to #4089

dberenbaum · 2022-12-06T19:19:56Z

content/docs/command-reference/update.md

+  For stages created with `dvc import-url` and a cloud-versioned URL, `--rev`
+  can be used to specify a object version ID to use. By default, the import will
+  be updated to the latest version from cloud storage.
+


Minor: link back to the --version-aware flag in import-url?

shcheklein · 2022-12-21T17:55:40Z

@dberenbaum @pmrowla qq - what is the status of this, and the cloud versioning in general?

dberenbaum · 2022-12-21T20:58:52Z

what is the status of this, and the cloud versioning in general?

It's on my plate to review and merge the outstanding docs PRs (here and #4165, and tracked more broadly in #4089). We discussed in sprint meeting and agreed we could merge now with an admon that it's experimental.

In Q1, the plan is to address performance issues and publicly promote it. Initial thoughts on blog post are in Notion.

per #4142 (review)

* guide: draft structure of Data Mgmt and some updates around the topic in existing docs * guide: full text for draft intro to DM * guide: hide cloud versioning info per #4042 (review) * guide: clarify Data Mgmt parts and add prospective figure titles * guide: add figure drafts to Data Mgmt * guide: SCM->VC (Data Mgmt) * guide: update 2 figs and add 1 more (Data Mgmt) * guide: roll back unrelated changes per #4042 (review) * guide: mention clouds first (DM) and and update fig. 1 per #4042 (review) * guide: flatten DM index per #4042 (review) * guide: udpates to DM/ DV moved from #4053 (review) * guide: add DM/ Data Versioning page per #4042 (comment) * guide: update outdated link * guide: revert more unrelatedly chaqnged files per #4042 (review) * guide: remove unused ref link * guide: DM/ Remote Storage (not just Setup) and and some links from cmd refs and avoid term "data remote" and some admons nearby... * guide: remove a comment * guide: draft for DM/ Remote Storage content * ref: expand config.remote and link to/from Remotes guide * ref: fix remote config file examples * guide: complete Remote Config section and and add Project config section to DM/ DV guide * ref: rewrite remote add and modify Descs * guide: complete list of supported storage types * ref: rewrite remote index page from extracted from #4053 * guide: clarify `remote modify` phrase in in the Remote config section of DM/ Remote Storage * Update content/docs/user-guide/data-management/data-versioning.md * guide: update versioning config per #4058 (review) * guide: don't call remote storage "additional" here (in the DM/ Remote Storage guide) per #4058 (review) Co-authored-by: Dave Berenbaum <[email protected]> * guide: pull -> download (DM/ RS intro) * guide: remove "optional" from Remote Storage nav & title per #4058 (review) * guide: splits and notes around Data Mgmt index page rel. #4042 (comment) * guide: Data Mgmt intro + note updates * guide: draft of all contents + + remove comments * guide: small impros to Data Mgmt in prep for #4042 (review) * guide: rewrite Data Mgmt index in before/after form per #4042 (review) * guide: add draft figure for Data Mgmt * guide: simplify/refocus data mgmt index per #4042 (review) * work around commented header bug * guide: drop DM/ DV page * guide: rewrite DM intro and - hide benefits (for now) - remove codification comment block * guide: use DM table instead of figure for now * guide: rewrite Data Mgmt story * guide: add draft figures to Data Mgmt * guide: simplify Data Mgmt story and benefits * guide: remove unused images (DM) * guide: update Data Mgmt figures (v1) * guide: rewrite text of Data Mgmt index * guide: update Data Mgmt figures * guide: iterate on Data Mgmt again * guide: update Data Mgmt figs * guide: more supporting info about Data Mgmt * guide: update figures (much more concrete) and and matching text updates * guide: edits to How it works (Data Mgmt) * guide: update Data Mgmt figures Rel. #4042 (comment) * guide: emphaisze dataset versions in UG fig 1 Rel. #4042 (comment) * guide: update Data Mgmt figures (with notes), expand img captions, and update text accordingly. * guide: more updates to text and figure styles, esp. to the first half and comment some stuff out (temporary) * guide: update figures and text (Data Mgmt) ... Using a tabs toggle for the 2nd fig. * guide: Data Management text (section 1) finalized for this version of figures * guide: Data Management (main text) finalized for this version of figures * guide: Data Management (secondary text) pending diagram and code sample(s) * guide: add DVC data mgmt technical diagram & dummy sample CLI blocks * guide: update Data Mgmt text * guide: udpate text and 2nd figure (Data Mgmt) * guide: draft 2nd and 3rd figures * guide: rewrite Data Mgmt/ How it works & and Benefits/ Tradeoffs Probably still unfinished... Missing more data versioning info? See HTML comments. * guide: update drafts of Data Mgmt figures 2, 3 * guide: Data Mgmt improvements and hide the benefits list for now * guide: separate from Data Mgmt work Rel. #4042 * Apply suggestions from code review * Merge branch main + * ref: bring cloud versioning copy edits of import-url from https://github.com/iterative/dvc.org/pull/4260/files#diff-ef95e18c4bd039757695065a23946dc27e28b4727ce07c670cdc096e34dbe3b3 * ref: clarify import-url with cloud versioning per #4142 (review) * ref: updates to import-url --version-aware and update --rev * ref: add import-url --version aware to Synopsis per #4089 (comment) * Restyled by prettier (#4266) Co-authored-by: Restyled.io <[email protected]> * ref: updates around worktree updates (cloud versioning) * ref: link from `remote` (index) to storage types * guide: roll back changes to dvc.yaml `rev` field spec * Update content/docs/command-reference/update.md * guide: link refs in .dvc file spec * Restyled by prettier (#4319) Co-authored-by: Restyled.io <[email protected]> * Update content/docs/command-reference/update.md --------- Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]>

* guide: draft structure of Data Mgmt and some updates around the topic in existing docs * guide: full text for draft intro to DM * guide: hide cloud versioning info per #4042 (review) * guide: clarify Data Mgmt parts and add prospective figure titles * guide: add figure drafts to Data Mgmt * guide: SCM->VC (Data Mgmt) * guide: update 2 figs and add 1 more (Data Mgmt) * guide: roll back unrelated changes per #4042 (review) * guide: mention clouds first (DM) and and update fig. 1 per #4042 (review) * guide: flatten DM index per #4042 (review) * guide: udpates to DM/ DV moved from #4053 (review) * guide: add DM/ Data Versioning page per #4042 (comment) * guide: update outdated link * guide: revert more unrelatedly chaqnged files per #4042 (review) * guide: remove unused ref link * guide: DM/ Remote Storage (not just Setup) and and some links from cmd refs and avoid term "data remote" and some admons nearby... * guide: remove a comment * guide: draft for DM/ Remote Storage content * ref: expand config.remote and link to/from Remotes guide * ref: fix remote config file examples * guide: complete Remote Config section and and add Project config section to DM/ DV guide * ref: rewrite remote add and modify Descs * guide: complete list of supported storage types * ref: rewrite remote index page from extracted from #4053 * guide: clarify `remote modify` phrase in in the Remote config section of DM/ Remote Storage * Update content/docs/user-guide/data-management/data-versioning.md * guide: update versioning config per #4058 (review) * guide: don't call remote storage "additional" here (in the DM/ Remote Storage guide) per #4058 (review) Co-authored-by: Dave Berenbaum <[email protected]> * guide: pull -> download (DM/ RS intro) * guide: remove "optional" from Remote Storage nav & title per #4058 (review) * guide: splits and notes around Data Mgmt index page rel. #4042 (comment) * guide: Data Mgmt intro + note updates * guide: draft of all contents + + remove comments * guide: small impros to Data Mgmt in prep for #4042 (review) * guide: rewrite Data Mgmt index in before/after form per #4042 (review) * guide: add draft figure for Data Mgmt * guide: simplify/refocus data mgmt index per #4042 (review) * work around commented header bug * guide: drop DM/ DV page * guide: rewrite DM intro and - hide benefits (for now) - remove codification comment block * guide: use DM table instead of figure for now * guide: rewrite Data Mgmt story * guide: add draft figures to Data Mgmt * guide: simplify Data Mgmt story and benefits * guide: remove unused images (DM) * guide: update Data Mgmt figures (v1) * guide: rewrite text of Data Mgmt index * guide: update Data Mgmt figures * guide: iterate on Data Mgmt again * guide: update Data Mgmt figs * guide: more supporting info about Data Mgmt * guide: update figures (much more concrete) and and matching text updates * guide: edits to How it works (Data Mgmt) * guide: update Data Mgmt figures Rel. #4042 (comment) * guide: emphaisze dataset versions in UG fig 1 Rel. #4042 (comment) * guide: update Data Mgmt figures (with notes), expand img captions, and update text accordingly. * guide: more updates to text and figure styles, esp. to the first half and comment some stuff out (temporary) * guide: update figures and text (Data Mgmt) ... Using a tabs toggle for the 2nd fig. * guide: Data Management text (section 1) finalized for this version of figures * guide: Data Management (main text) finalized for this version of figures * guide: Data Management (secondary text) pending diagram and code sample(s) * guide: add DVC data mgmt technical diagram & dummy sample CLI blocks * guide: update Data Mgmt text * guide: udpate text and 2nd figure (Data Mgmt) * guide: draft 2nd and 3rd figures * guide: rewrite Data Mgmt/ How it works & and Benefits/ Tradeoffs Probably still unfinished... Missing more data versioning info? See HTML comments. * guide: update drafts of Data Mgmt figures 2, 3 * guide: Data Mgmt improvements and hide the benefits list for now * guide: separate from Data Mgmt work Rel. #4042 * Apply suggestions from code review * Merge branch main + * ref: update links from API to Remotes guide * guide: update links around Remote Storage and and other updates to nearby Markdown (e.g. proper admons) * Roll back unrelated changes * Restyled by prettier (#4261) Co-authored-by: Restyled.io <[email protected]> * ref: bring cloud versioning copy edits of import-url from https://github.com/iterative/dvc.org/pull/4260/files#diff-ef95e18c4bd039757695065a23946dc27e28b4727ce07c670cdc096e34dbe3b3 * ref: clarify import-url with cloud versioning per #4142 (review) * ref: updates to import-url --version-aware and update --rev * ref: add import-url --version aware to Synopsis per #4089 (comment) * Restyled by prettier (#4266) Co-authored-by: Restyled.io <[email protected]> * Restyled by prettier (#4322) Co-authored-by: Restyled.io <[email protected]> * Update content/docs/command-reference/remote/modify.md Co-authored-by: Oded Messer <[email protected]> * Update content/docs/command-reference/remote/modify.md Co-authored-by: Oded Messer <[email protected]> * Update content/docs/command-reference/push.md Co-authored-by: Oded Messer <[email protected]> * yarn format-all --------- Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: rogermparent <[email protected]> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Oded Messer <[email protected]>

pmrowla added 2 commits November 29, 2022 19:24

ref: document import-url --version-aware

a87198b

ref: document push field for dvc files

b1a2349

pmrowla self-assigned this Nov 29, 2022

pmrowla requested a review from dberenbaum November 29, 2022 10:29

shcheklein temporarily deployed to dvc-org-import-url-vers-fuikfw November 29, 2022 10:29 Inactive

dberenbaum reviewed Nov 29, 2022

View reviewed changes

pmrowla mentioned this pull request Dec 2, 2022

import-url: disable push by default for cloud-versioned imports iterative/dvc#8578

Merged

2 tasks

jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) C: ref Content of /doc/*-reference labels Dec 3, 2022

jorgeorpinel mentioned this pull request Dec 3, 2022

worktree: support push: false iterative/dvc#8581

Merged

2 tasks

jorgeorpinel reviewed Dec 3, 2022

View reviewed changes

content/docs/user-guide/project-structure/dvc-files.md Outdated Show resolved Hide resolved

content/docs/user-guide/project-structure/dvcyaml-files.md Outdated Show resolved Hide resolved

guide: push metafile field text update

fccda92

restyled-io bot mentioned this pull request Dec 3, 2022

Restyle ref: document import-url cloud versioning changes #4158

Merged

Restyled by prettier (#4158)

60df383

Co-authored-by: Restyled.io <[email protected]>

shcheklein temporarily deployed to dvc-org-import-url-vers-fuikfw December 3, 2022 23:21 Inactive

jorgeorpinel reviewed Dec 3, 2022

View reviewed changes

restyled-io bot mentioned this pull request Dec 6, 2022

Restyle ref: document import-url cloud versioning changes #4164

Closed

shcheklein temporarily deployed to dvc-org-import-url-vers-fuikfw December 6, 2022 06:40 Inactive

add information on supported storages

e3255f2

pmrowla force-pushed the import-url-versioning branch from fbb8a47 to e3255f2 Compare December 6, 2022 08:42

shcheklein temporarily deployed to dvc-org-import-url-vers-fuikfw December 6, 2022 08:42 Inactive

ref: document update for versioned import-url

d5a4e9a

shcheklein temporarily deployed to dvc-org-import-url-vers-fuikfw December 6, 2022 10:33 Inactive

dberenbaum reviewed Dec 6, 2022

View reviewed changes

dberenbaum approved these changes Dec 6, 2022

View reviewed changes

add link to import-url --version-aware

2d3e24a

shcheklein temporarily deployed to dvc-org-import-url-vers-nbrzm5 December 9, 2022 04:43 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-tcz3nx December 9, 2022 04:55 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-xtrkfr December 9, 2022 05:09 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-ftuefl December 9, 2022 05:21 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-ihsb0r December 9, 2022 05:35 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-wtvwhs December 9, 2022 05:48 Inactive

shcheklein temporarily deployed to dvc-org-import-url-vers-6ietae December 9, 2022 06:01 Inactive

dberenbaum merged commit 90fea91 into main Dec 22, 2022

dberenbaum deleted the import-url-versioning branch December 22, 2022 19:53

dberenbaum mentioned this pull request Dec 22, 2022

Cloud versioning (initial guide and ref updates) #4165

Merged

jorgeorpinel mentioned this pull request Dec 23, 2022

guide: Data Versioning #4119

Closed

1 task

jorgeorpinel added a commit that referenced this pull request Jan 26, 2023

ref: clarify import-url with cloud versioning

4fec3d2

per #4142 (review)

This was referenced Jan 26, 2023

ML Pipelines: recommend push=false for intermediate outputs #4267

Closed

ref: remote updates #4062

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref: document `import-url` cloud versioning changes #4142

ref: document `import-url` cloud versioning changes #4142

pmrowla commented Nov 29, 2022 •

edited by jorgeorpinel

Loading

github-actions bot commented Nov 29, 2022 •

edited

Loading

dberenbaum Nov 29, 2022

jorgeorpinel Dec 3, 2022 •

edited

Loading

jorgeorpinel Dec 3, 2022

jorgeorpinel left a comment

jorgeorpinel Dec 3, 2022 •

edited

Loading

pmrowla Dec 6, 2022 •

edited

Loading

dberenbaum Dec 7, 2022

dberenbaum commented Dec 5, 2022

dberenbaum Dec 6, 2022

shcheklein commented Dec 21, 2022

dberenbaum commented Dec 21, 2022

ref: document import-url cloud versioning changes #4142

ref: document import-url cloud versioning changes #4142

Conversation

pmrowla commented Nov 29, 2022 • edited by jorgeorpinel Loading

github-actions bot commented Nov 29, 2022 • edited Loading

Link Check Report

dberenbaum Nov 29, 2022

Choose a reason for hiding this comment

jorgeorpinel Dec 3, 2022 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Dec 3, 2022

Choose a reason for hiding this comment

jorgeorpinel left a comment

Choose a reason for hiding this comment

jorgeorpinel Dec 3, 2022 • edited Loading

Choose a reason for hiding this comment

pmrowla Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

dberenbaum Dec 7, 2022

Choose a reason for hiding this comment

dberenbaum commented Dec 5, 2022

dberenbaum Dec 6, 2022

Choose a reason for hiding this comment

shcheklein commented Dec 21, 2022

dberenbaum commented Dec 21, 2022

ref: document `import-url` cloud versioning changes #4142

ref: document `import-url` cloud versioning changes #4142

pmrowla commented Nov 29, 2022 •

edited by jorgeorpinel

Loading

github-actions bot commented Nov 29, 2022 •

edited

Loading

jorgeorpinel Dec 3, 2022 •

edited

Loading

jorgeorpinel Dec 3, 2022 •

edited

Loading

pmrowla Dec 6, 2022 •

edited

Loading