ref: create Remote Reference (config) #4264

jorgeorpinel · 2023-01-26T08:00:31Z

Addresses the main part of #2866:

Currently most of the information about remote storage is in the command reference, mainly under the remote add/modify commands (which have expandable details sections for each supported remote type. We can leave some basics in there (cmd refs should be self-contained), but the details need to be in the User Guide instead. Probably as part of #2856

Maybe also

There should be some basic info and linking from the config guide into the remotes guide though.

In review app: https://dvc-org-guide-data-mgmt-leaijc.herokuapp.com/doc/user-guide/data-management/remote-storage

AFTER:

Co-authored-by: Restyled.io <[email protected]>

github-actions · 2023-01-26T08:14:03Z

Link Check Report

All 34 links passed!

jorgeorpinel · 2023-01-26T09:58:41Z

Hi @dberenbaum, me again...

So far this only creates the first of a proposed page-per-remote Ref section (could be renamed or moved to a wider Configuration Ref). This was pretty-labor intensive so let's review this one first to decide whether to move fwd with this strategy. Thanks, PTAL when you can:

~~Changes to remote add ref.: Shortened and un-hid the S3 section.~~
~~Changes to remote modify ref.: simplified the S3 section a lot (but still long enough to stay hidden)~~
~~New Amazon S3 remote ref page with all the details in a more readable/guided presentation~~

Cc @daavoo @casperdcl

jorgeorpinel · 2023-01-26T18:16:27Z

Discussed a bit with @shcheklein and we agreed this is a good direction (long overdue) but also to further simplify cmd refs and to bury these pages under the Data Management guide for now (we expect readers to find them via search results or links probably)... ⌛

daavoo · 2023-01-26T18:53:17Z

So far this only creates the first of a proposed page-per-remote Ref section (could be renamed or moved to a wider Configuration Ref). This was pretty-labor intensive so let's review this one first to decide whether to move fwd with this strategy. Thanks, PTAL when you can:

I really like the changes and the separated page.
TBH I would even be more aggressive and remove any s3 specific info (i.e. Example: Customize an S3 remote section) from remote/add remote/modify pages in favor of just a link to the s3 specific page.

dberenbaum · 2023-01-26T22:17:56Z

Yup, direction looks good! Thank you @jorgeorpinel!

dberenbaum · 2023-01-26T22:19:37Z

Changes to remote modify ref.: simplified the S3 section a lot (but still long enough to stay hidden)

Agree with @daavoo that if we go this direction, I don't think we should keep this part of the cmd ref at all.

burry these pages under the Data Management guide for now

Not sure I follow. What pages do you want to bury? I really like the top-level remote ref section you added.

Edit: if it's getting to be too many top-level sections, we could have one top-level "Reference" section with subsections like "Command," "Python API," "Remote", "File."

and some typo fixes

and remove repeated content from cmd refs (link to guide)

jorgeorpinel · 2023-02-01T20:49:18Z

I don't think we should keep this part of the cmd ref at all.

OK, I'm just listing the remote types in 3 places then: the Remote Storage guide index page, the remote add ref, and the remote modify ref (WIP).

What pages do you want to bury?

To avoid introducing a new top-level section on the docs nav, we are burying the specific storage type pages under User Guide -> Data Management -> Remote Storage -> each page (e.g. S3). Not expecting people to navigate there by clicking the nav (but rather links or search results), this should be fine 🐶🔥.

jorgeorpinel

@dberenbaum should we review this now? The PR is starting to grow too much. Maybe we can merge it in with just the first type in and I'll then make another PR for the remaining ones. Cc @shcheklein

Also, this PR already has a child to update related links (#4282)...

List of changes:

jorgeorpinel · 2023-02-02T05:27:46Z

content/docs/sidebar.json

-          "remote-storage",
+          {
+            "slug": "remote-storage",
+            "source": "remote-storage/index.md",
+            "children": ["amazon-s3"]


Made this into a section (moved remote-storage.md to remote-storage/index.md)

jorgeorpinel · 2023-02-02T05:29:29Z

content/docs/user-guide/data-management/remote-storage/amazon-s3.md

+# Amazon S3
+
+`dvc remote add` a (new) remote name and a valid [S3] URL:
+
+```cli
+$ dvc remote add -d myremote s3://<bucket>/<key>


Created Amazon S3 page, we can use as template for the remaining types.

jorgeorpinel · 2023-02-02T05:32:26Z

content/docs/user-guide/data-management/remote-storage/amazon-s3.md

+## S3-compatible servers (non-Amazon)
+
+Set the `endpointurl` parameter with the URL to connect to the S3-compatible
+service (e.g. [MinIO], [IBM Cloud Object Storage], etc.). For example, let's set
+up a [DigitalOcean Space] (equivalent to a bucket in S3) called `mystore` found


The Amazon S3 page includes the non-Amazon, S3-compatible section, since it only involves one config option.

jorgeorpinel · 2023-02-02T05:32:58Z

content/docs/command-reference/remote/add.md

-### Amazon S3
-
-> 💡 Before adding an S3 remote, be sure to
-> [Create a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html).
-
-```cli
-$ dvc remote add -d myremote s3://mybucket/path


S3 info is now totally removed (just linked instead) from remote add...

jorgeorpinel · 2023-02-02T05:33:28Z

content/docs/command-reference/remote/add.md

-### S3-compatible storage
-
-For object storage that supports an S3-compatible API (e.g.
-[Minio](https://min.io/),
-[DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/),


(including S3-compatible)

jorgeorpinel · 2023-02-02T05:33:48Z

content/docs/command-reference/remote/modify.md

-### Amazon S3
-
- `url` - remote location, in the `s3://<bucket>/<key>` format:
-
-  ```cli
-  $ dvc remote modify myremote url s3://mybucket/path


...and from remote modify

jorgeorpinel · 2023-02-02T05:34:14Z

content/docs/command-reference/remote/modify.md

-### S3-compatible storage
+Each type of storage has different config options you can set. See all the
+details in the pages linked below.

- `endpointurl` - URL to connect to the S3-compatible storage server or service
-  (e.g. [Minio](https://min.io/),
-  [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/),


(including S3-compatible as well)

jorgeorpinel · 2023-02-02T05:36:19Z

content/docs/user-guide/data-management/remote-storage/index.md

+## File systems (local remotes)
+
+<admon type="tip">
+
+Not related to the `--local` option of `dvc remote` and `dvc config`!
+
+</admon>


This file is the same as what's now in https://dvc.org/doc/user-guide/data-management/remote-storage (I guess I moved it in separate commits so the Git diff isn't clear),

except for this new section on local remotes, which (like S3-compatible) I'm not sure require their own page.

The rest of the changes are mainly links and admonitions around this new section and other changes here.

Note there's also a nested PR specifically for link updates as well: #4282

update admonitions and links

github-actions · 2023-02-23T04:30:40Z

Link Check Report

There were no links to check!

jorgeorpinel · 2023-02-23T04:31:24Z

Is there any discussion you can point me to where you and @shcheklein explain why it would be too much to add it to the top-level or as another reference type? I liked the previous version at the top level since it's one of the first things almost all users need to configure.

Sorry @dberenbaum, it was during a call but the logic is pretty much what I wrote. Either way it's a simple change to make but it may be best to decide before merging to avoid having to make redirects soon after.

Can we finalize and merge this one?

I've solved conflicts for now so this is ready for merge, I think.

…s3.md

per #4264 (comment)

per #4264 (review)

dberenbaum

Thanks @jorgeorpinel! Added a couple small edits to clarify, especially while we are in limbo trying to transfer all the ref info to guides. As long as it looks okay with those changes, I'll go ahead and merge.

content/docs/user-guide/data-management/remote-storage/index.md

content/docs/user-guide/data-management/remote-storage/amazon-s3.md

…s3.md

Co-authored-by: Restyled.io <[email protected]>

content/docs/user-guide/data-management/remote-storage/index.md

* ref: start Remote Reference (config) * Restyled by prettier (#4265) Co-authored-by: Restyled.io <[email protected]> * guide: move Remote Storage ref into Data Mgmt * start: links to new Remotes guide and and some typo fixes * guide: finalize S3 storage page and and remove repeated content from cmd refs (link to guide) * guide: link to remote storage types in link * guide: move "local remotes" to Remotes (index page) and update admonitions and links * ref: remove S3 examples * guide: emphasize that remotes use regular cloud storage config * Update content/docs/user-guide/data-management/remote-storage/amazon-s3.md * guide: drop `worktree` cloud versioning from Remotes Config per #4264 (comment) * guide: move cloud versioning near the top of Remote Config per #4264 (review) --------- Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Dave Berenbaum <[email protected]>

jorgeorpinel · 2023-02-23T20:02:15Z

LGTM

* ref: start Remote Reference (config) * Restyled by prettier (#4265) Co-authored-by: Restyled.io <[email protected]> * guide: move Remote Storage ref into Data Mgmt * start: links to new Remotes guide and and some typo fixes * guide: finalize S3 storage page and and remove repeated content from cmd refs (link to guide) * guide: move "local remotes" to Remotes (index page) and update admonitions and links * ref: remove S3 examples * guide: Azure remote page and start GCS * guide: finish GCS page and improvements to the other ones (S3, Azure) * guide: small link fix in GDrive how-to * guide: emphasize that remotes use regular cloud storage config * Update content/docs/user-guide/data-management/remote-storage/amazon-s3.md * guide: drop `worktree` cloud versioning from Remotes Config per #4264 (comment) * guide: move cloud versioning near the top of Remote Config per #4264 (review) * fix a link * typo * reformat all storage types (Data Mgmt/ Remote Storage) * guide: move admon about pending Remote guides up rel. #4284 (review) * link all remote types (instead of admon) per #4284 (review) * Restyled by prettier (#4333) Co-authored-by: Restyled.io <[email protected]> * Update content/docs/user-guide/data-management/remote-storage/amazon-s3.md Co-authored-by: Jorge Orpinel <[email protected]> --------- Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Dave Berenbaum <[email protected]>

ref: start Remote Reference (config)

659828f

jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) p1-important Active priorities to deal within next sprints C: ref Content of /doc/*-reference labels Jan 26, 2023

restyled-io bot mentioned this pull request Jan 26, 2023

Restyle ref: create Remote Reference (config) #4265

Merged

Restyled by prettier (#4265)

d49afa6

Co-authored-by: Restyled.io <[email protected]>

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc January 26, 2023 08:02 Inactive

jorgeorpinel mentioned this pull request Jan 26, 2023

Cloud versioning (initial guide and ref updates) #4165

Merged

Merge branch 'main' into guide/data-mgmt/remote-config

12430a6

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc January 27, 2023 05:45 Inactive

jorgeorpinel added 2 commits January 27, 2023 17:14

Merge branch 'main' into guide/data-mgmt/remote-config

a91e7c5

guide: move Remote Storage ref into Data Mgmt

bef1201

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 1, 2023 16:16 Inactive

start: links to new Remotes guide and

27bdcd2

and some typo fixes

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 1, 2023 17:16 Inactive

guide: finalize S3 storage page and

b2cc99c

and remove repeated content from cmd refs (link to guide)

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 1, 2023 20:43 Inactive

jorgeorpinel mentioned this pull request Feb 1, 2023

guide: link to remote storage types in link #4282

Merged

1 task

jorgeorpinel added the C: guide Content of /doc/user-guide label Feb 1, 2023

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 2, 2023 05:23 Inactive

jorgeorpinel commented Feb 2, 2023

View reviewed changes

guide: move "local remotes" to Remotes (index page) and

94a16c1

update admonitions and links

jorgeorpinel force-pushed the guide/data-mgmt/remote-config branch from d07f2bf to 94a16c1 Compare February 2, 2023 05:38

Merge branch 'main' into guide/data-mgmt/remote-config

033ecc8

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 04:21 Inactive

restyled-io bot mentioned this pull request Feb 23, 2023

Restyle ref: create Remote Reference (config) #4329

Closed

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 04:34 Inactive

Update content/docs/user-guide/data-management/remote-storage/amazon-…

2badd87

…s3.md

jorgeorpinel force-pushed the guide/data-mgmt/remote-config branch from 80d251c to 2badd87 Compare February 23, 2023 04:34

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 04:35 Inactive

guide: drop worktree cloud versioning from Remotes Config

b9f3b1d

per #4264 (comment)

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 04:41 Inactive

guide: move cloud versioning near the top of Remote Config

acdebaa

per #4264 (review)

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 04:51 Inactive

jorgeorpinel added the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Feb 23, 2023

dberenbaum approved these changes Feb 23, 2023

View reviewed changes

content/docs/user-guide/data-management/remote-storage/index.md Show resolved Hide resolved

content/docs/user-guide/data-management/remote-storage/amazon-s3.md Outdated Show resolved Hide resolved

Dave Berenbaum added 2 commits February 23, 2023 11:11

Update content/docs/user-guide/data-management/remote-storage/amazon-…

f9fb502

…s3.md

Update content/docs/user-guide/data-management/remote-storage/index.md

1523b84

restyled-io bot mentioned this pull request Feb 23, 2023

Restyle ref: create Remote Reference (config) #4331

Merged

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 16:12 Inactive

Restyled by prettier (#4331)

8a01e64

Co-authored-by: Restyled.io <[email protected]>

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 17:02 Inactive

dberenbaum reviewed Feb 23, 2023

View reviewed changes

content/docs/user-guide/data-management/remote-storage/index.md Outdated Show resolved Hide resolved

Update content/docs/user-guide/data-management/remote-storage/index.md

2a25526

shcheklein temporarily deployed to dvc-org-guide-data-mgmt-leaijc February 23, 2023 17:12 Inactive

dberenbaum merged commit 9bf61df into main Feb 23, 2023

dberenbaum deleted the guide/data-mgmt/remote-config branch February 23, 2023 18:25

jorgeorpinel removed the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref: create Remote Reference (config) #4264

ref: create Remote Reference (config) #4264

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

github-actions bot commented Jan 26, 2023 •

edited

Loading

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

daavoo commented Jan 26, 2023 •

edited

Loading

dberenbaum commented Jan 26, 2023

dberenbaum commented Jan 26, 2023 •

edited

Loading

jorgeorpinel commented Feb 1, 2023

jorgeorpinel left a comment •

edited

Loading

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

jorgeorpinel Feb 2, 2023

github-actions bot commented Feb 23, 2023 •

edited

Loading

jorgeorpinel commented Feb 23, 2023

dberenbaum left a comment

jorgeorpinel commented Feb 23, 2023

ref: create Remote Reference (config) #4264

ref: create Remote Reference (config) #4264

Conversation

jorgeorpinel commented Jan 26, 2023 • edited Loading

github-actions bot commented Jan 26, 2023 • edited Loading

Link Check Report

jorgeorpinel commented Jan 26, 2023 • edited Loading

jorgeorpinel commented Jan 26, 2023 • edited Loading

daavoo commented Jan 26, 2023 • edited Loading

dberenbaum commented Jan 26, 2023

dberenbaum commented Jan 26, 2023 • edited Loading

jorgeorpinel commented Feb 1, 2023

jorgeorpinel left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 23, 2023 • edited Loading

Link Check Report

jorgeorpinel commented Feb 23, 2023

dberenbaum left a comment

Choose a reason for hiding this comment

jorgeorpinel commented Feb 23, 2023

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

github-actions bot commented Jan 26, 2023 •

edited

Loading

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

jorgeorpinel commented Jan 26, 2023 •

edited

Loading

daavoo commented Jan 26, 2023 •

edited

Loading

dberenbaum commented Jan 26, 2023 •

edited

Loading

jorgeorpinel left a comment •

edited

Loading

github-actions bot commented Feb 23, 2023 •

edited

Loading