Skip to content

Commit

Permalink
Update docs for remote_store.moving_average_window_size
Browse files Browse the repository at this point in the history
Signed-off-by: Bhumika Saini <[email protected]>
  • Loading branch information
Bhumika Saini committed Sep 11, 2023
1 parent 0e2a588 commit 8471d24
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 5 deletions.
1 change: 1 addition & 0 deletions _api-reference/cluster-api/cluster-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ The following request field parameters are compatible with the cluster API.
| cluster.max_shards_per_node | Integer | Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. Default is `1000`. |
| cluster.persistent_tasks.allocation.enable | String | Enables or disables allocation for persistent tasks: <br /> <br /> `all` – Allows persistent tasks to be assigned to nodes. <br /> <br /> `none` – No allocations are allowed for persistent tasks. This does not affect persistent tasks already running. <br /> <br /> Default is `all`. |
| cluster.persistent_tasks.allocation.recheck_interval | Time unit | The cluster manager automatically checks whether or not persistent tasks need to be assigned when the cluster state changes in a significant way. There are other factors, such as memory usage, that will affect whether or not persistent tasks are assigned to nodes but do not otherwise cause the cluster state to change. This setting defines how often assignment checks are performed in response to these factors. Default is `30 seconds`, with a minimum of `10 seconds` being required. |
| remote_store.moving_average_window_size | Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |

#### Example request

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Remote segment backpressure is a shard-level rejection mechanism that dynamicall
Remote segment backpressure is activated if any of the following thresholds is breached:

- **Consecutive failure**: The backpressure is activated if there are _N_ or more consecutive failures. The value of _N_ is configurable in `remote_store.segment.pressure.consecutive_failures.limit`.
- **Bytes lag**: The bytes lag is computed by adding the size of all files that are present in local committed segments but not in the remote store. The backpressure is activated if the bytes lag is greater than _K_ multiplied by the moving average of the size (in bytes) of files uploaded after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.bytes_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_bytes_moving_average_window_size`.
- **Time lag**: The time lag is computed by comparing the timestamps of the most recent local refresh and the most recent remote store segment upload. The backpressure is activated if the time lag is greater than _K_ multiplied by the moving average of the time taken to upload new segments and metadata file after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.time_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_time_moving_average_window_size`.
- **Bytes lag**: The bytes lag is computed by adding the size of all files that are present in local committed segments but not in the remote store. The backpressure is activated if the bytes lag is greater than _K_ multiplied by the moving average of the size (in bytes) of files uploaded after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.bytes_lag.variance_factor`. The moving window size is configurable in `remote_store.moving_average_window_size`.
- **Time lag**: The time lag is computed by comparing the timestamps of the most recent local refresh and the most recent remote store segment upload. The backpressure is activated if the time lag is greater than _K_ multiplied by the moving average of the time taken to upload new segments and metadata file after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.time_lag.variance_factor`. The moving window size is configurable in `remote_store.moving_average_window_size`.

## Handling segment merges

Expand All @@ -36,13 +36,11 @@ The following table lists the settings used for activating backpressure. For thr
|`remote_store.segment.pressure.enabled` |Boolean | If `true`, enables remote segment backpressure. Default is `false`. |
|`remote_store.segment.pressure.consecutive_failures.limit` |Integer |The minimum consecutive failure count for activating remote segment backpressure. Default is `5`. |
|`remote_store.segment.pressure.bytes_lag.variance_factor` |Float | The variance factor that is used together with the moving average to calculate the dynamic bytes lag threshold for activating remote segment backpressure. Default is `10`. |
|`remote_store.segment.pressure.upload_bytes_moving_average_window_size` |Integer |The moving average window size that is used to calculate the dynamic bytes lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. |
|`remote_store.segment.pressure.time_lag.variance_factor` |Float |The variance factor that is used together with the moving average to calculate the dynamic time lag threshold for activating remote segment backpressure. Default is `10`. |
|`remote_store.segment.pressure.upload_time_moving_average_window_size` |Integer |The moving average window size that is used to calculate the dynamic time lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`.|

The following table lists the settings used for statistics.

|Setting |Data type |Description |
|:--- |:--- |:--- |
|`remote_store.segment.pressure.upload_bytes_per_sec_moving_average_window_size` |Integer |The moving average window size, which is exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`.|
|`remote_store.moving_average_window_size` |Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |

0 comments on commit 8471d24

Please sign in to comment.