Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain disk headroom settings more in docs #90763

Merged
merged 1 commit into from
Oct 20, 2022
Merged

Explain disk headroom settings more in docs #90763

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 36 additions & 9 deletions docs/reference/modules/cluster/disk_allocator.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,15 +107,6 @@ is now `true`. The setting will be removed in a future release.
(<<dynamic-cluster-setting,Dynamic>>)
Controls the flood stage watermark, which defaults to 95%. {es} enforces a read-only index block (`index.blocks.read_only_allow_delete`) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., `0.95`, or an absolute byte value.

`cluster.routing.allocation.disk.watermark.flood_stage.max_headroom`::
(<<dynamic-cluster-setting,Dynamic>>) Controls the max headroom for the flood stage watermark (in case of a percentage/ratio value).
Defaults to 100GB when
`cluster.routing.allocation.disk.watermark.flood_stage` is not explicitly set.
This caps the amount of free space required.

NOTE: You cannot mix the usage of percentage/ratio values and byte values within
the watermark settings. Either all values are set to percentage/ratio values, or all are set to byte values. This enforcement is so that {es} can validate that the settings are internally consistent, ensuring that the low disk threshold is less than the high disk threshold, and the high disk threshold is less than the flood stage threshold. A similar check is done for the max headroom values.

An example of resetting the read-only index block on the `my-index-000001` index:

[source,console]
Expand All @@ -129,6 +120,21 @@ PUT /my-index-000001/_settings
--
// end::cluster-routing-flood-stage-tag[]

`cluster.routing.allocation.disk.watermark.flood_stage.max_headroom`::
(<<dynamic-cluster-setting,Dynamic>>) Controls the max headroom for the flood stage watermark (in case of a percentage/ratio value).
Defaults to 100GB when
`cluster.routing.allocation.disk.watermark.flood_stage` is not explicitly set.
This caps the amount of free space required.

NOTE: You cannot mix the usage of percentage/ratio values and byte values across
the `cluster.routing.allocation.disk.watermark.low`, `cluster.routing.allocation.disk.watermark.high`,
and `cluster.routing.allocation.disk.watermark.flood_stage` settings. Either all values
are set to percentage/ratio values, or all are set to byte values. This enforcement is
so that {es} can validate that the settings are internally consistent, ensuring that the
low disk threshold is less than the high disk threshold, and the high disk threshold is
less than the flood stage threshold. A similar comparison check is done for the max
headroom values.

[[cluster-routing-flood-stage-frozen]]
// tag::cluster-routing-flood-stage-tag[]
`cluster.routing.allocation.disk.watermark.flood_stage.frozen` {ess-icon}::
Expand Down Expand Up @@ -169,3 +175,24 @@ PUT _cluster/settings
}
}
--------------------------------------------------

Concerning the max headroom settings for the watermarks, please note
that these apply only in the case that the watermark settings are percentages/ratios.
The aim of a max headroom value is to cap the required free disk space before hitting
the respective watermark. This is especially useful for servers with larger
disks, where a percentage/ratio watermark could translate to a big free disk space requirement,
and the max headroom can be used to cap the required free disk space amount.
As an example, let us take the default settings for the flood watermark.
It has a 95% default value, and the flood max headroom setting has a default value of 100GB.
This means that:

* For a smaller disk, e.g., of 100GB, the flood watermark will hit at 95%, meaning at 5GB
of free space, since 5GB is smaller than the 100GB max headroom value.
* For a larger disk, e.g., of 100TB, the flood watermark will hit at 100GB of free space.
That is because the 95% flood watermark alone would require 5TB of free disk space, but
that is capped by the max headroom setting to 100GB.

Finally, the max headroom settings have their default values only if their respective watermark
settings are not explicitly set (thus, they have their default percentage values).
If watermarks are explicitly set, then the max headroom settings do not have their default values,
and would need to be explicitly set if they are desired.