Skip to content

Commit

Permalink
Explain disk headroom settings more in docs
Browse files Browse the repository at this point in the history
Relates to elastic#81406
  • Loading branch information
kingherc committed Oct 10, 2022
1 parent b3775ee commit 981f592
Showing 1 changed file with 36 additions and 9 deletions.
45 changes: 36 additions & 9 deletions docs/reference/modules/cluster/disk_allocator.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -107,15 +107,6 @@ is now `true`. The setting will be removed in a future release.
(<<dynamic-cluster-setting,Dynamic>>)
Controls the flood stage watermark, which defaults to 95%. {es} enforces a read-only index block (`index.blocks.read_only_allow_delete`) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., `0.95`, or an absolute byte value.

`cluster.routing.allocation.disk.watermark.flood_stage.max_headroom`::
(<<dynamic-cluster-setting,Dynamic>>) Controls the max headroom for the flood stage watermark (in case of a percentage/ratio value).
Defaults to 100GB when
`cluster.routing.allocation.disk.watermark.flood_stage` is not explicitly set.
This caps the amount of free space required.

NOTE: You cannot mix the usage of percentage/ratio values and byte values within
the watermark settings. Either all values are set to percentage/ratio values, or all are set to byte values. This enforcement is so that {es} can validate that the settings are internally consistent, ensuring that the low disk threshold is less than the high disk threshold, and the high disk threshold is less than the flood stage threshold. A similar check is done for the max headroom values.

An example of resetting the read-only index block on the `my-index-000001` index:

[source,console]
Expand All @@ -129,6 +120,21 @@ PUT /my-index-000001/_settings
--
// end::cluster-routing-flood-stage-tag[]

`cluster.routing.allocation.disk.watermark.flood_stage.max_headroom`::
(<<dynamic-cluster-setting,Dynamic>>) Controls the max headroom for the flood stage watermark (in case of a percentage/ratio value).
Defaults to 100GB when
`cluster.routing.allocation.disk.watermark.flood_stage` is not explicitly set.
This caps the amount of free space required.

NOTE: You cannot mix the usage of percentage/ratio values and byte values across
the `cluster.routing.allocation.disk.watermark.low`, `cluster.routing.allocation.disk.watermark.high`,
and `cluster.routing.allocation.disk.watermark.flood_stage` settings. Either all values
are set to percentage/ratio values, or all are set to byte values. This enforcement is
so that {es} can validate that the settings are internally consistent, ensuring that the
low disk threshold is less than the high disk threshold, and the high disk threshold is
less than the flood stage threshold. A similar comparison check is done for the max
headroom values.

[[cluster-routing-flood-stage-frozen]]
// tag::cluster-routing-flood-stage-tag[]
`cluster.routing.allocation.disk.watermark.flood_stage.frozen` {ess-icon}::
Expand Down Expand Up @@ -169,3 +175,24 @@ PUT _cluster/settings
}
}
--------------------------------------------------

Concerning the max headroom settings for the watermarks, please note
that these apply only in the case that the watermark settings are percentages/ratios.
The aim of a max headroom value is to cap the required free disk space before hitting
the respective watermark. This is especially useful for servers with larger
disks, where a percentage/ratio watermark could translate to a big free disk space requirement,
and the max headroom can be used to cap the required free disk space amount.
As an example, let us take the default settings for the flood watermark.
It has a 95% default value, and the flood max headroom setting has a default value of 100GB.
This means that:

* For a smaller disk, e.g., of 100GB, the flood watermark will hit at 95%, meaning at 5GB
of free space, since 5GB is smaller than the 100GB max headroom value.
* For a larger disk, e.g., of 100TB, the flood watermark will hit at 100GB of free space.
That is because the 95% flood watermark alone would require 5TB of free disk space, but
that is capped by the max headroom setting to 100GB.

Finally, the max headroom settings have their default values only if their respective watermark
settings are not explicitly set (thus, they have their default percentage values).
If watermarks are explicitly set, then the max headroom settings do not have their default values,
and would need to be explicitly set if they are desired.

0 comments on commit 981f592

Please sign in to comment.