Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs for Remote Segment and Remote Translog store stats surfaced via nodes/indices/cluster stats APIs #4995

Merged
merged 7 commits into from
Sep 13, 2023
Merged

Update docs for Remote Segment and Remote Translog store stats surfaced via nodes/indices/cluster stats APIs #4995

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8471d24
Update docs for remote_store.moving_average_window_size
Sep 8, 2023
afad4e0
Update docs for Remote Segment and Remote Translog store stats surfac…
Sep 8, 2023
3fb096d
Apply suggestions from code review
Naarcha-AWS Sep 13, 2023
3ddba4f
Update _api-reference/nodes-apis/nodes-stats.md
Naarcha-AWS Sep 13, 2023
de043b3
Update nodes-stats.md
Naarcha-AWS Sep 13, 2023
c7dcd37
Update cluster-stats.md
Naarcha-AWS Sep 13, 2023
fff4b08
Apply suggestions from code review
Naarcha-AWS Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _api-reference/cluster-api/cluster-settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ The following request field parameters are compatible with the cluster API.
| cluster.max_shards_per_node | Integer | Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. Default is `1000`. |
| cluster.persistent_tasks.allocation.enable | String | Enables or disables allocation for persistent tasks: <br /> <br /> `all` – Allows persistent tasks to be assigned to nodes. <br /> <br /> `none` – No allocations are allowed for persistent tasks. This does not affect persistent tasks already running. <br /> <br /> Default is `all`. |
| cluster.persistent_tasks.allocation.recheck_interval | Time unit | The cluster manager automatically checks whether or not persistent tasks need to be assigned when the cluster state changes in a significant way. There are other factors, such as memory usage, that will affect whether or not persistent tasks are assigned to nodes but do not otherwise cause the cluster state to change. This setting defines how often assignment checks are performed in response to these factors. Default is `30 seconds`, with a minimum of `10 seconds` being required. |
| remote_store.moving_average_window_size | Integer | The moving average window size used to calculate the rolling statistic values exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`. Minimum enforced is `5`. |
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

#### Example request

Expand Down
23 changes: 23 additions & 0 deletions _api-reference/cluster-api/cluster-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,29 @@ Parameter | Type | Description
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 1112,
"max_unsafe_auto_id_timestamp": 1644269449096,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"mappings": {
Expand Down
156 changes: 152 additions & 4 deletions _api-reference/index-apis/stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,14 +199,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -326,14 +363,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -457,14 +531,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down Expand Up @@ -584,14 +695,51 @@ By default, the returned statistics are aggregated in the `primaries` and `total
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes": {}
},
"translog": {
"operations": 0,
"size_in_bytes": 55,
"uncommitted_operations": 0,
"uncommitted_size_in_bytes": 55,
"earliest_last_modified_age": 142622215
"earliest_last_modified_age": 142622215,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache": {
"memory_size_in_bytes": 0,
Expand Down
66 changes: 65 additions & 1 deletion _api-reference/nodes-apis/nodes-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,14 +245,51 @@ Select the arrow to view the example response.
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 288,
"max_unsafe_auto_id_timestamp" : -1,
"remote_store" : {
"upload" : {
"total_upload_size" : {
"started_bytes" : 152419,
"succeeded_bytes" : 152419,
"failed_bytes" : 0
},
"refresh_size_lag" : {
"total_bytes" : 0,
"max_bytes" : 0
},
"max_refresh_time_lag_in_millis" : 0,
"total_time_spent_in_millis" : 516
},
"download" : {
"total_download_size" : {
"started_bytes" : 0,
"succeeded_bytes" : 0,
"failed_bytes" : 0
},
"total_time_spent_in_millis" : 0
}
},
"file_sizes" : { }
},
"translog" : {
"operations" : 12,
"size_in_bytes" : 1452,
"uncommitted_operations" : 12,
"uncommitted_size_in_bytes" : 1452,
"earliest_last_modified_age" : 164160
"earliest_last_modified_age" : 164160,
"remote_store" : {
"upload" : {
"total_uploads" : {
"started" : 57,
"failed" : 0,
"succeeded" : 57
},
"total_upload_size" : {
"started_bytes" : 16830,
"failed_bytes" : 0,
"succeeded_bytes" : 16830
}
}
}
},
"request_cache" : {
"memory_size_in_bytes" : 1649,
Expand Down Expand Up @@ -792,13 +829,40 @@ segments.index_writer_memory_in_bytes | Integer | The total amount of memory use
segments.version_map_memory_in_bytes | Integer | The total amount of memory used by all version maps, in bytes.
segments.fixed_bit_set_memory_in_bytes | Integer | The total amount of memory used by fixed bit sets, in bytes. Fixed bit sets are used for nested objects and join fields.
segments.max_unsafe_auto_id_timestamp | Integer | The timestamp for the most recently retired indexing request, in milliseconds since the epoch.
segments.remote_store | Object | Statistics about Remote Segment Store operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload | Object | Statistics about Remote Segment Store upload operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size | Object | The amount of data uploaded to Remote Segment Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes for which upload to Remote Segment Store has been started.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes for which upload to Remote Segment Store has succeeded.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes for which upload to Remote Segment Store has failed.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.refresh_size_lag | Object | Statistics about the bytes lag between the Remote Segment Store and local store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.refresh_size_lag.total_bytes | Integer | The total bytes lag between the Remote Segment Store and local store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.refresh_size_lag.max_bytes | Integer | The maximum bytes lag between the Remote Segment Store and local store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.max_refresh_time_lag_in_millis | Integer | The max time duration (in milliseconds) the remote refresh is behind the local refresh.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.upload.total_time_spent_in_millis | Integer | The total time duration (in milliseconds) spent on uploads to the Remote Segment Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download | Object | Statistics about Remote Segment Store download operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size | Object | The amount of data downloaded from Remote Segment Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size.started_bytes | Integer | The number of bytes for which download from Remote Segment Store has been started.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size.succeeded_bytes | Integer | The number of bytes for which download from Remote Segment Store has succeeded.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_download_size.failed_bytes | Integer | The number of bytes for which download from Remote Segment Store has failed.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.remote_store.download.total_time_spent_in_millis | Integer | The total time duration (in milliseconds) spent on downloads from the Remote Segment Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
segments.file_sizes | Integer | Statistics about the size of the segment files.
translog | Object | Statistics about transaction log operations for the node.
translog.operations | Integer | The number of translog operations.
translog.size_in_bytes | Integer | The size of the translog, in bytes.
translog.uncommitted_operations | Integer | The number of uncommitted translog operations.
translog.uncommitted_size_in_bytes | Integer | The size of uncommitted translog operations, in bytes.
translog.earliest_last_modified_age | Integer | The earliest last modified age for the translog.
translog.remote_store | Object | Statistics about Remote Translog Store operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload | Object | Statistics about Remote Translog Store upload operations.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_uploads | Object | The number of syncs to Remote Translog Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_uploads.started | Integer | The number of syncs to Remote Translog Store that have been started.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_uploads.failed | Integer | The number of syncs to Remote Translog Store that have failed.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_uploads.succeeded | Integer | The number of syncs to Remote Translog Store that have succeeded.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_upload_size | Object | The amount of data uploaded to Remote Translog Store.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_upload_size.started_bytes | Integer | The number of bytes for which upload to Remote Translog Store has been started.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_upload_size.failed_bytes | Integer | The number of bytes for which upload to Remote Translog Store has failed.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
translog.remote_store.upload.total_upload_size.succeeded_bytes | Integer | The number of bytes for which upload to Remote Translog Store has succeeded.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
request_cache | Object | Statistics about the request cache for the node.
request_cache.memory_size_in_bytes | Integer | The memory size used by the request cache, in bytes.
request_cache.evictions | Integer | The number of request cache evictions.
Expand Down
Loading