Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Remove soft limit for snapshot repositories #80745

Merged
merged 5 commits into from
Nov 16, 2021
Merged

[DOCS] Remove soft limit for snapshot repositories #80745

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1defe91
[DOCS] Remove 200 snapshot soft limit for repo
jrodewig Nov 15, 2021
521dbb8
Re-add retention guidance
jrodewig Nov 16, 2021
bbe793c
reword
jrodewig Nov 16, 2021
d5d4d6b
remove disk space
jrodewig Nov 16, 2021
afc4c5a
reword
jrodewig Nov 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions docs/Versions.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,3 @@ ifeval::["{source_branch}"=="7.x"]
:apm-server-ref-v: {apm-server-ref-m}
:apm-overview-ref-v: {apm-overview-ref-m}
endif::[]

// Max recommended snapshots in a snapshot repo.
// Used in the snapshot/restore docs.
:max-snapshot-count: 200
4 changes: 0 additions & 4 deletions docs/reference/slm/apis/slm-put.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,13 @@ Time period after which a snapshot is considered expired and eligible for
deletion. {slm-init} deletes expired snapshots based on the
<<slm-retention-schedule,`slm.retention_schedule`>>.

// To update {max-snapshot-count}, see docs/Version.asciidoc
`max_count`::
(Optional, integer)
Maximum number of snapshots to retain, even if the snapshots have not yet
expired. If the number of snapshots in the repository exceeds this limit, the
policy retains the most recent snapshots and deletes older snapshots. This limit
only includes snapshots with a <<get-snapshot-api-response-state,`state`>> of
`SUCCESS`.
+
NOTE: This value should not exceed {max-snapshot-count}. See
<<snapshot-retention-limits>>.

`min_count`::
(Optional, integer)
Expand Down
15 changes: 8 additions & 7 deletions docs/reference/snapshot-restore/take-snapshot.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,10 +272,13 @@ policy. Other snapshots don't count toward the policy's retention limits.
[[snapshot-retention-limits]]
==== Snapshot retention limits

While not a hard limit, a snapshot repository shouldn't contain more than
{max-snapshot-count} snapshots at a time. This ensures the repository's metadata
doesn't grow to a size that may destabilize the master node. We recommend you
set up your {slm-init} policy's retention rules to enforce this limit.
Comment on lines -275 to -278
Copy link
Contributor Author

@jrodewig jrodewig Nov 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@original-brownbear Let me know if we want to offer any guidance to users for sizing their snapshot repositories. I can also use this information to update the policy UI copy in Kibana.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say we shouldn't give a hard number here these days probably. Maybe we should just point out that the repository does scale to thousands of snapshots, but that larger repositories do require more memory/a larger master. Just to make it clear that people should have retention setup in SLM and can't outright run a snapshot every 5 minutes for the next 10 years without ever deleting anything?
As long as we at least make it clear that there are limits and master node costs here and that deletes are necessary eventually we're good I think.

Copy link
Contributor Author

@jrodewig jrodewig Nov 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I re-added some guidance based on your comment with 521dbb8. Let me know what you think.

If that looks good to you, I'll merge this and get a fix open for the Kibana UI.

We recommend you include retention rules in your {slm-init} policy to delete
snapshots you no longer need.

A snapshot repository can safely scale to thousands of snapshots. However, to
manage its metadata, a large repository requires more memory and disk space on
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"disk space" is not true, sorry that was my unfortunate wording above. It's really just memory that's the limiting factor here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the correction.

the master node. Retention rules ensure a repository's metadata doesn't grow to
a size that may destabilize the master node.

[discrete]
[[manually-create-snapshot]]
Expand Down Expand Up @@ -536,9 +539,7 @@ from a previous week or month.
To fix this, you can create multiple {slm-init} policies with the same snapshot
repository that run on different schedules. Since a policy's retention rules
only apply to its snapshots, a policy won't delete a snapshot created by another
policy. However, you'll need to ensure the total number of snapshots in the
repository doesn't exceed the <<snapshot-retention-limits,{max-snapshot-count}
snapshot soft limit>>.
policy.

For example, the following {slm-init} policy takes hourly snapshots with a
maximum of 24 snapshots. The policy keeps its snapshots for one day.
Expand Down