From 8471d24113b8e979b88c7cf9b91eb686e5892b4a Mon Sep 17 00:00:00 2001 From: Bhumika Saini Date: Fri, 8 Sep 2023 15:41:02 +0530 Subject: [PATCH] Update docs for remote_store.moving_average_window_size Signed-off-by: Bhumika Saini --- _api-reference/cluster-api/cluster-settings.md | 1 + .../remote-store/remote-segment-backpressure.md | 8 +++----- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/_api-reference/cluster-api/cluster-settings.md b/_api-reference/cluster-api/cluster-settings.md index 4a8639cbc0..49c7475ecf 100644 --- a/_api-reference/cluster-api/cluster-settings.md +++ b/_api-reference/cluster-api/cluster-settings.md @@ -107,6 +107,7 @@ The following request field parameters are compatible with the cluster API. | cluster.max_shards_per_node | Integer | Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. Default is `1000`. | | cluster.persistent_tasks.allocation.enable | String | Enables or disables allocation for persistent tasks:

`all` – Allows persistent tasks to be assigned to nodes.

`none` – No allocations are allowed for persistent tasks. This does not affect persistent tasks already running.

Default is `all`. | | cluster.persistent_tasks.allocation.recheck_interval | Time unit | The cluster manager automatically checks whether or not persistent tasks need to be assigned when the cluster state changes in a significant way. There are other factors, such as memory usage, that will affect whether or not persistent tasks are assigned to nodes but do not otherwise cause the cluster state to change. This setting defines how often assignment checks are performed in response to these factors. Default is `30 seconds`, with a minimum of `10 seconds` being required. | +| remote_store.moving_average_window_size | Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. | #### Example request diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md index 4fa427509a..707bb6ee29 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md @@ -18,8 +18,8 @@ Remote segment backpressure is a shard-level rejection mechanism that dynamicall Remote segment backpressure is activated if any of the following thresholds is breached: - **Consecutive failure**: The backpressure is activated if there are _N_ or more consecutive failures. The value of _N_ is configurable in `remote_store.segment.pressure.consecutive_failures.limit`. -- **Bytes lag**: The bytes lag is computed by adding the size of all files that are present in local committed segments but not in the remote store. The backpressure is activated if the bytes lag is greater than _K_ multiplied by the moving average of the size (in bytes) of files uploaded after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.bytes_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_bytes_moving_average_window_size`. -- **Time lag**: The time lag is computed by comparing the timestamps of the most recent local refresh and the most recent remote store segment upload. The backpressure is activated if the time lag is greater than _K_ multiplied by the moving average of the time taken to upload new segments and metadata file after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.time_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_time_moving_average_window_size`. +- **Bytes lag**: The bytes lag is computed by adding the size of all files that are present in local committed segments but not in the remote store. The backpressure is activated if the bytes lag is greater than _K_ multiplied by the moving average of the size (in bytes) of files uploaded after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.bytes_lag.variance_factor`. The moving window size is configurable in `remote_store.moving_average_window_size`. +- **Time lag**: The time lag is computed by comparing the timestamps of the most recent local refresh and the most recent remote store segment upload. The backpressure is activated if the time lag is greater than _K_ multiplied by the moving average of the time taken to upload new segments and metadata file after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.time_lag.variance_factor`. The moving window size is configurable in `remote_store.moving_average_window_size`. ## Handling segment merges @@ -36,13 +36,11 @@ The following table lists the settings used for activating backpressure. For thr |`remote_store.segment.pressure.enabled` |Boolean | If `true`, enables remote segment backpressure. Default is `false`. | |`remote_store.segment.pressure.consecutive_failures.limit` |Integer |The minimum consecutive failure count for activating remote segment backpressure. Default is `5`. | |`remote_store.segment.pressure.bytes_lag.variance_factor` |Float | The variance factor that is used together with the moving average to calculate the dynamic bytes lag threshold for activating remote segment backpressure. Default is `10`. | -|`remote_store.segment.pressure.upload_bytes_moving_average_window_size` |Integer |The moving average window size that is used to calculate the dynamic bytes lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. | |`remote_store.segment.pressure.time_lag.variance_factor` |Float |The variance factor that is used together with the moving average to calculate the dynamic time lag threshold for activating remote segment backpressure. Default is `10`. | -|`remote_store.segment.pressure.upload_time_moving_average_window_size` |Integer |The moving average window size that is used to calculate the dynamic time lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`.| The following table lists the settings used for statistics. |Setting |Data type |Description | |:--- |:--- |:--- | -|`remote_store.segment.pressure.upload_bytes_per_sec_moving_average_window_size` |Integer |The moving average window size, which is exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`.| +|`remote_store.moving_average_window_size` |Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |