Skip to content

Commit

Permalink
Update docs for Remote Segment and Remote Translog store stats surfac…
Browse files Browse the repository at this point in the history 
…ed via nodes/indices/cluster stats APIs (#4995)

* Update docs for remote_store.moving_average_window_size

Signed-off-by: Bhumika Saini <[email protected]>

* Update docs for Remote Segment and Remote Translog store stats surfaced via nodes/indices/cluster stats APIs

Signed-off-by: Bhumika Saini <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _api-reference/nodes-apis/nodes-stats.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update nodes-stats.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update cluster-stats.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Bhumika Saini <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
  • Loading branch information
@Naarcha-AWS @natebower @vagimeli
3 people authored and vagimeli committed Sep 19, 2023
1 parent 3726eb1 commit fa64141
Show file tree
Hide file tree
Showing 5 changed files with 249 additions and 10 deletions.
1 change: 1 addition & 0 deletions _api-reference/cluster-api/cluster-settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ The following request field parameters are compatible with the cluster API.
| cluster.max_shards_per_node | Integer | Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. Default is `1000`. |
| cluster.persistent_tasks.allocation.enable | String | Enables or disables allocation for persistent tasks: <br /> <br /> `all` – Allows persistent tasks to be assigned to nodes. <br /> <br /> `none` – No allocations are allowed for persistent tasks. This does not affect persistent tasks already running. <br /> <br /> Default is `all`. |
| cluster.persistent_tasks.allocation.recheck_interval | Time unit | The cluster manager automatically checks whether or not persistent tasks need to be assigned when the cluster state changes in a significant way. There are other factors, such as memory usage, that will affect whether or not persistent tasks are assigned to nodes but do not otherwise cause the cluster state to change. This setting defines how often assignment checks are performed in response to these factors. Default is `30 seconds`, with a minimum of `10 seconds` being required. |
| remote_store.moving_average_window_size | Integer | The moving average window size used to calculate the rolling statistic values exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |

#### Example request

Expand Down
24 changes: 24 additions & 0 deletions _api-reference/cluster-api/cluster-stats.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,29 @@ Parameter | Type | Description
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 1112,
"max_unsafe_auto_id_timestamp": 1644269449096,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"mappings": {
Expand Down Expand Up @@ -491,3 +514,4 @@ nodes.network_types | The transport and HTTP networks within the nodes.
nodes.discovery_type | The method the nodes use to find other nodes within the cluster.
nodes.packaging_types | Information about the nodes' OpenSearch distribution.
nodes.ingest | Information about the nodes' ingest pipelines/nodes, if there are any.
total_time_spent | The total amount of download and upload time spent across all shards in the cluster when downloading or uploading from the remote store.
156 changes: 152 additions & 4 deletions _api-reference/index-apis/stats.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -199,14 +199,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -326,14 +363,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -457,14 +531,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -584,14 +695,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down
70 changes: 69 additions & 1 deletion _api-reference/nodes-apis/nodes-stats.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,14 +245,51 @@ Select the arrow to view the example response.
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 288,
"max_unsafe_auto_id_timestamp" : -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes" : { }
},
"translog" : {
"operations" : 12,
"size_in_bytes" : 1452,
"uncommitted_operations" : 12,
"uncommitted_size_in_bytes" : 1452,
"earliest_last_modified_age" : 164160
"earliest_last_modified_age" : 164160,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache" : {
"memory_size_in_bytes" : 1649,
Expand Down Expand Up @@ -792,13 +829,44 @@ segments.index_writer_memory_in_bytes | Integer | The total amount of memory use
segments.version_map_memory_in_bytes | Integer | The total amount of memory used by all version maps, in bytes.
segments.fixed_bit_set_memory_in_bytes | Integer | The total amount of memory used by fixed bit sets, in bytes. Fixed bit sets are used for nested objects and join fields.
segments.max_unsafe_auto_id_timestamp | Integer | The timestamp for the most recently retired indexing request, in milliseconds since the epoch.
segments.segment_replication | Object | Segment replication statistics for all primary shards when segment replication is enabled on the node.
segments.segment_replication.maxBytesBehind | long | The maximum number of bytes behind the primary replica.
segments.segment_replication.totalBytesBehind | long | The total number of bytes behind the primary replicas.
segments.segment_replication.maxReplicationLag | long | The maximum amount of time, in milliseconds, taken by a replica to catch up to its primary.
segments.remote_store | Object | Statistics about remote segment store operations.
segments.remote_store.upload | Object | Statistics related to uploads to the remote segment store.
segments.remote_store.upload.total_upload_size | Object | The amount of data, in bytes, uploaded to the remote segment store.
segments.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes to upload to the remote segment store after the upload has started.
segments.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes successfully uploaded to the remote segment store.
segments.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes that failed to upload to the remote segment store.
segments.remote_store.upload.refresh_size_lag | Object | The amount of lag during upload between the remote segment store and the local store.
segments.remote_store.upload.refresh_size_lag.total_bytes | Integer | The total number of bytes that lagged during the upload refresh between the remote segment store and the local store.
segments.remote_store.upload.refresh_size_lag.max_bytes | Integer | The maximum amount of lag, in bytes, during the upload refresh between the remote segment store and the local store.
segments.remote_store.upload.max_refresh_time_lag_in_millis | Integer | The maximum duration, in milliseconds, that the remote refresh is behind the local refresh.
segments.remote_store.upload.total_time_spent_in_millis | Integer | The total amount of time, in milliseconds, spent on uploads to the remote segment store.
segments.remote_store.download | Object | Statistics related to downloads to the remote segment store.
segments.remote_store.download.total_download_size | Object | The total amount of data download from the remote segment store.
segments.remote_store.download.total_download_size.started_bytes | Integer | The number of bytes downloaded from the remote segment store after the download starts.
segments.remote_store.download.total_download_size.succeeded_bytes | Integer | The number of bytes successfully downloaded from the remote segment store.
segments.remote_store.download.total_download_size.failed_bytes | Integer | The number of bytes that failed to download from the remote segment store.
segments.remote_store.download.total_time_spent_in_millis | Integer | The total duration, in milliseconds, spent on downloads from the remote segment store.
segments.file_sizes | Integer | Statistics about the size of the segment files.
translog | Object | Statistics about transaction log operations for the node.
translog.operations | Integer | The number of translog operations.
translog.size_in_bytes | Integer | The size of the translog, in bytes.
translog.uncommitted_operations | Integer | The number of uncommitted translog operations.
translog.uncommitted_size_in_bytes | Integer | The size of uncommitted translog operations, in bytes.
translog.earliest_last_modified_age | Integer | The earliest last modified age for the translog.
translog.remote_store | Object | Statistics related to operations from the remote translog store.
translog.remote_store.upload | Object | Statistics related to uploads to the remote translog store.
translog.remote_store.upload.total_uploads | Object | The number of syncs to the remote translog store.
translog.remote_store.upload.total_uploads.started | Integer | The number of upload syncs to the remote translog store that have started.
translog.remote_store.upload.total_uploads.failed | Integer | The number of failed upload syncs to the remote translog store.
translog.remote_store.upload.total_uploads.succeeded | Integer | The number of successful upload syncs to the remote translog store.
translog.remote_store.upload.total_upload_size | Object | The total amount of data uploaded to the remote translog store.
translog.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes actively uploading to the remote translog store after the upload has started.
translog.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes that failed to upload to the remote translog store.
translog.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes successfully uploaded to the remote translog store.
request_cache | Object | Statistics about the request cache for the node.
request_cache.memory_size_in_bytes | Integer | The memory size used by the request cache, in bytes.
request_cache.evictions | Integer | The number of request cache evictions.
Expand Down
Loading

0 comments on commit fa64141

Please sign in to comment.