Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommend NFS over S3-like repositories #107297

Merged
merged 1 commit into from
Apr 10, 2024

Conversation

DaveCTurner
Copy link
Contributor

Allegedly-S3-compatible APIs are very popular these days, but many
third-party systems offering such an API also support a shared
filesystem interface. Shared filesystem protocols such as NFS are much
better specified than the S3 API, and experience shows that they lead to
fewer compatibility headaches. This commit adds a recommendation to the
repository-s3 docs to consider such an interface instead.

Allegedly-S3-compatible APIs are very popular these days, but many
third-party systems offering such an API also support a shared
filesystem interface. Shared filesystem protocols such as NFS are much
better specified than the S3 API, and experience shows that they lead to
fewer compatibility headaches. This commit adds a recommendation to the
`repository-s3` docs to consider such an interface instead.
@DaveCTurner DaveCTurner added >docs General docs changes :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.14.0 v8.13.3 labels Apr 10, 2024
Copy link
Contributor

Documentation preview:

@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Apr 10, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added the Team:Docs Meta label for docs team label Apr 10, 2024
@DaveCTurner DaveCTurner requested a review from ywangd April 10, 2024 07:53
Copy link
Member

@ywangd ywangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines +546 to +548
behaviour in full. If you are using such a system for your snapshots, consider
using a <<snapshots-filesystem-repository,shared filesystem repository>> based
on a standardized protocol such as NFS to access your storage system instead.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to have a bit softer statement to say something like "If you are using such a system for snapshots, especially if such system does not pass verify snapshot repository tests, consider ..."? Is it helpful at all? Or maybe it is more misleading since the verify API may adapot new S3 features and what was compatible may become incompatible later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the "consider" is soft enough. If the system doesn't pass verification then it absolutely shouldn't be used, but yeah we've seen enough systems that pass today but then fail some stricter checks in a future version. I think NFS is always worth considering in these situations, it just doesn't have the same problems.

@DaveCTurner DaveCTurner merged commit ceeee1b into elastic:main Apr 10, 2024
6 checks passed
@DaveCTurner DaveCTurner deleted the 2024/04/10/nfs-over-s3 branch April 10, 2024 10:39
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.13

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request Apr 10, 2024
Allegedly-S3-compatible APIs are very popular these days, but many
third-party systems offering such an API also support a shared
filesystem interface. Shared filesystem protocols such as NFS are much
better specified than the S3 API, and experience shows that they lead to
fewer compatibility headaches. This commit adds a recommendation to the
`repository-s3` docs to consider such an interface instead.
elasticsearchmachine pushed a commit that referenced this pull request Apr 10, 2024
Allegedly-S3-compatible APIs are very popular these days, but many
third-party systems offering such an API also support a shared
filesystem interface. Shared filesystem protocols such as NFS are much
better specified than the S3 API, and experience shows that they lead to
fewer compatibility headaches. This commit adds a recommendation to the
`repository-s3` docs to consider such an interface instead.
@DaveCTurner DaveCTurner restored the 2024/04/10/nfs-over-s3 branch June 17, 2024 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >docs General docs changes Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Docs Meta label for docs team v8.13.3 v8.14.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants