Skip to content

Commit

Permalink
Merge branch 'main' into add-dashboards-assistant-doc
Browse files Browse the repository at this point in the history
  • Loading branch information
ruanyl authored Oct 31, 2024
2 parents 8f4d230 + bd1631e commit ed84563
Show file tree
Hide file tree
Showing 55 changed files with 3,223 additions and 45 deletions.
3 changes: 1 addition & 2 deletions .github/ISSUE_TEMPLATE/issue_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ assignees: ''

**Tell us about your request.** Provide a summary of the request.

***Version:** List the OpenSearch version to which this issue applies, e.g. 2.14, 2.12--2.14, or all.
**Version:** List the OpenSearch version to which this issue applies, e.g. 2.14, 2.12--2.14, or all.

**What other resources are available?** Provide links to related issues, POCs, steps for testing, etc.

111 changes: 80 additions & 31 deletions _api-reference/cluster-api/cluster-stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ The cluster stats API operation returns statistics about your cluster.
```json
GET _cluster/stats
GET _cluster/stats/nodes/<node-filters>
GET _cluster/stats/<metric>/nodes/<node-filters>
GET _cluster/stats/<metric>/<index_metric>/nodes/<node-filters>
```

## Path parameters
Expand All @@ -30,10 +32,57 @@ All parameters are optional.
Parameter | Type | Description
:--- | :--- | :---
&lt;node-filters&gt; | List | A comma-separated list of [node filters]({{site.url}}{{site.baseurl}}/api-reference/nodes-apis/index/#node-filters) that OpenSearch uses to filter results.
metric | String | A comma-separated list of [metric groups](#metric-groups), for example, `jvm,fs`. Default is all metric groups.
index_metric | String | A comma-separated list of [index metric groups](#index-metric-groups), for example, `docs,store`. Default is all index metrics.


Although the `master` node is now called `cluster_manager` for version 2.0, we retained the `master` field for backwards compatibility. If you have a node that has either a `master` role or a `cluster_manager` role, the `count` increases for both fields by 1. To see an example node count increase, see the Response sample.
{: .note }
Although the term `master` was deprecated in favor of `cluster_manager` subsequent to OpenSearch 2.0, the `master` field was retained for backward compatibility. If you have a node that has either a `master` role or a `cluster_manager` role, the `count` increases for both fields by 1. For an example node count increase, see the [example response](#example-response).
{: .note }

### Metric groups

The following table lists all available metric groups.

Metric | Description
:--- |:----
`indices` | Statistics about indexes in the cluster.
`os` | Statistics about the host OS, including load and memory.
`process` | Statistics about processes, including open file descriptors and CPU usage.
`jvm` | Statistics about the JVM, including heap usage and threads.
`fs` | Statistics about file system usage.
`plugins` | Statistics about OpenSearch plugins integrated with the nodes.
`network_types` | A list of the transport and HTTP networks connected to the nodes.
`discovery_type` | The method used by the nodes to find other nodes in the cluster.
`packaging_types` | Information about each node's OpenSearch distribution.
`ingest` | Statistics about ingest pipelines.

### Index metric groups

To filter the information returned for the `indices` metric, you can use specific `index_metric` values. These values are only supported when using the following query types:

```json
GET _cluster/stats/_all/<index_metric>/_nodes/<node-filters>
GET _cluster/stats/indices/<index_metric>/_nodes/<node-filters>
```

The following index metrics are supported:

- `shards`
- `docs`
- `store`
- `fielddata`
- `query_cache`
- `completion`
- `segments`
- `mappings`
- `analysis`

For example, the following query requests statistics for `docs` and `search`:

```json
GET _cluster/stats/indices/docs,segments/_nodes/_all
```
{% include copy-curl.html %}

## Example request

Expand Down Expand Up @@ -491,32 +540,32 @@ GET _cluster/stats/nodes/_cluster_manager

Field | Description
:--- | :---
nodes | How many nodes returned in the response.
cluster_name | The cluster's name.
cluster_uuid | The cluster's uuid.
timestamp | The Unix epoch time of when the cluster was last refreshed.
status | The cluster's health status.
indices | Statistics about the indexes in the cluster.
indices.count | How many indexes are in the cluster.
indices.shards | Information about the cluster's shards.
indices.docs | How many documents are still in the cluster and how many documents are deleted.
indices.store | Information about the cluster's storage.
indices.fielddata | Information about the cluster's field data
indices.query_cache | Data about the cluster's query cache.
indices.completion | How many bytes in memory are used to complete operations.
indices.segments | Information about the cluster's segments, which are small Lucene indexes.
indices.mappings | Mappings within the cluster.
indices.analysis | Information about analyzers used in the cluster.
nodes | Statistics about the nodes in the cluster.
nodes.count | How many nodes were returned from the request.
nodes.versions | OpenSearch's version number.
nodes.os | Information about the operating systems used in the nodes.
nodes.process | The processes the returned nodes use.
nodes.jvm | Statistics about the Java Virtual Machines in use.
nodes.fs | The nodes' file storage.
nodes.plugins | The OpenSearch plugins integrated within the nodes.
nodes.network_types | The transport and HTTP networks within the nodes.
nodes.discovery_type | The method the nodes use to find other nodes within the cluster.
nodes.packaging_types | Information about the nodes' OpenSearch distribution.
nodes.ingest | Information about the nodes' ingest pipelines/nodes, if there are any.
total_time_spent | The total amount of download and upload time spent across all shards in the cluster when downloading or uploading from the remote store.
`nodes` | The number of nodes returned in the response.
`cluster_name` | The cluster's name.
`cluster_uuid` | The cluster's UUID.
`timestamp` | The Unix epoch time indicating when the cluster was last refreshed.
`status` | The cluster's health status.
`indices` | Statistics about the indexes in the cluster.
`indices.count` | The number of indexes in the cluster.
`indices.shards` | Information about the cluster's shards.
`indices.docs` | The number of documents remaining in the cluster and the number of documents that were deleted.
`indices.store` | Information about the cluster's storage.
`indices.fielddata` | Information about the cluster's field data.
`indices.query_cache` | Data about the cluster's query cache.
`indices.completion` | The number of bytes in memory that were used to complete operations.
`indices.segments` | Information about the cluster's segments, which are small Lucene indexes.
`indices.mappings` | Information about mappings in the cluster.
`indices.analysis` | Information about analyzers used in the cluster.
`nodes` | Statistics about the nodes in the cluster.
`nodes.count` | The number of nodes returned by the request.
`nodes.versions` | The OpenSearch version number for each node.
`nodes.os` | Information about the operating systems used by the nodes.
`nodes.process` | A list of processes used by each node.
`nodes.jvm` | Statistics about the JVMs in use.
`nodes.fs` | Information about the node's file storage.
`nodes.plugins` | A list of the OpenSearch plugins integrated with the nodes.
`nodes.network_types` | A list of the transport and HTTP networks connected to the nodes.
`nodes.discovery_type` | A list of methods used by the nodes to find other nodes in the cluster.
`nodes.packaging_types` | Information about each node's OpenSearch distribution.
`nodes.ingest` | Information about the node's ingest pipelines/nodes, if there are any.
`total_time_spent` | The total amount of download and upload time spent across all shards in the cluster when downloading or uploading from the remote store.
175 changes: 175 additions & 0 deletions _api-reference/list/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
---
layout: default
title: List API
nav_order: 45
has_children: true
---

# List APIs
**Introduced 2.18**
{: .label .label-purple }

The List API retrieves statistics about indexes and shards in a paginated format. This streamlines the task of processing responses that include many indexes.

The List API supports two operations:

- [List indices]({{site.url}}{{site.baseurl}}/api-reference/list/list-indices/)
- [List shards]({{site.url}}{{site.baseurl}}/api-reference/list/list-shards/)

## Shared query parameters

All List API operations support the following optional query parameters.

Parameter | Description
:--- | :--- |
`v` | Provides verbose output by adding headers to the columns. It also adds some formatting to help align each of the columns. All examples in this section include the `v` parameter.
`help` | Lists the default and other available headers for a given operation.
`h` | Limits the output to specific headers.
`format` | The format in which to return the result. Valid values are `json`, `yaml`, `cbor`, and `smile`.
`s` | Sorts the output by the specified columns.

## Examples

The following examples show how to use the optional query parameters to customize all List API responses.


### Get verbose output

To query indexes and their statistics with a verbose output that includes all column headings in the response, use the `v` query parameter, as shown in the following example.

#### Request

```json
GET _list/indices?v
```
{% include copy-curl.html %}

#### Response

```json
health status index uuid pri rep docs.count docs.deleted
green open .kibana_1 - - - -
yellow open sample-index-1 - - - -
next_token null
```


### Get all available headers

To see all the available headers, use the `help` parameter with the following syntax:

```json
GET _list/<operation_name>?help
```
{% include copy-curl.html %}

#### Request

The following example list indices operation returns all the available headers:

```json
GET _list/indices?help
```
{% include copy-curl.html %}

#### Response

The following example displays the indexes and their health status in a table:

```json
health | h | current health status
status | s | open/close status
index | i,idx | index name
uuid | id,uuid | index uuid
pri | p,shards.primary,shardsPrimary | number of primary shards
rep | r,shards.replica,shardsReplica | number of replica shards
docs.count | dc,docsCount | available docs
```

### Get a subset of headers

To limit the output to a subset of headers, use the `h` parameter with the following syntax:

```json
GET _list/<operation_name>?h=<header_name_1>,<header_name_2>&v
```
{% include copy-curl.html %}

For any operation, you can determine which headers are available by using the `help` parameter and then using the `h` parameter to limit the output to only a subset of headers.

#### Request

The following example limits the indexes in the response to only the index name and health status headers:

```json
GET _list/indices?h=health,index
```
{% include copy-curl.html %}

### Response

```json
green .kibana_1
yellow sample-index-1
next_token null
```


### Sort by a header

To sort the output on a single page by a header, use the `s` parameter with the following syntax:

```json
GET _list/<operation_name>?s=<header_name_1>,<header_name_2>
```
{% include copy-curl.html %}

#### Request

The following example request sorts indexes by index name:

```json
GET _list/indices?s=h,i
```
{% include copy-curl.html %}

#### Response

```json
green sample-index-2
yellow sample-index-1
next_token null
```

### Retrieve data in JSON format

By default, List APIs return data in a `text/plain` format. Other supported formats are [YAML](https://yaml.org/), [CBOR](https://cbor.io/), and [Smile](https://github.com/FasterXML/smile-format-specification).


To retrieve data in the JSON format, use the `format=json` parameter with the following syntax.

If you use the Security plugin, ensure you have the appropriate permissions.
{: .note }

#### Request

```json
GET _list/<operation_name>?format=json
```
{% include copy-curl.html %}

#### Request

```json
GET _list/indices?format=json
```
{% include copy-curl.html %}

### Response

The response contains data in JSON format:

```json
{"next_token":null,"indices":[{"health":"green","status":"-","index":".kibana_1","uuid":"-","pri":"-","rep":"-","docs.count":"-","docs.deleted":"-","store.size":"-","pri.store.size":"-"},{"health":"yellow","status":"-","index":"sample-index-1","uuid":"-","pri":"-","rep":"-","docs.count":"-","docs.deleted":"-","store.size":"-","pri.store.size":"-"}]}
```

83 changes: 83 additions & 0 deletions _api-reference/list/list-indices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
layout: default
title: List indices
parent: List API
nav_order: 25
has_children: false
---

# List indices
**Introduced 2.18**
{: .label .label-purple }

The list indices operation provides the following index information in a paginated format:

- The amount of disk space used by the index.
- The number of shards contained in the index.
- The index's health status.

## Path and HTTP methods

```json
GET _list/indices
GET _list/indices/<index>
```

## Query parameters

Parameter | Type | Description
:--- | :--- | :---
`bytes` | Byte size | Specifies the units for the byte size, for example, `7kb` or `6gb`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).
`health` | String | Limits indexes based on their health status. Supported values are `green`, `yellow`, and `red`.
`include_unloaded_segments` | Boolean | Whether to include information from segments not loaded into memory. Default is `false`.
`cluster_manager_timeout` | Time | The amount of time to wait for a connection to the cluster manager node. Default is `30s`.
`pri` | Boolean | Whether to return information only from the primary shards. Default is `false`.
`time` | Time | Specifies the time units, for example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).
`expand_wildcards` | Enum | Expands wildcard expressions to concrete indexes. Combine multiple values with commas. Supported values are `all`, `open`, `closed`, `hidden`, and `none`. Default is `open`.
`next_token` | String | Fetches the next page of indexes. When `null`, only provides the first page of indexes. Default is `null`.
`size` | Integer | The maximum number of indexes to be displayed on a single page. The number of indexes on a single page of the response is not always equal to the specified `size`. Default is `500`. Minimum is `1` and maximum value is `5000`.
`sort` | String | The order in which the indexes are displayed. If `desc`, then the most recently created indexes are displayed first. If `asc`, then the oldest indexes are displayed first. Default is `asc`.

When using the `next_token` path parameter, use the token produced by the response to see the next page of indexes. After the API returns `null`, all indexes contained in the API have been returned.
{: .tip }


## Example requests

To get information for all the indexes, use the following query and keep specifying the `next_token` as received from response until its `null`:

```json
GET _list/indices/<index>?v&next_token=token
```


To limit the information to a specific index, add the index name after your query, as shown in the following example:

```json
GET _list/indices/<index>?v
```
{% include copy-curl.html %}

To get information about more than one index, separate the indexes with commas, as shown in the following example:

```json
GET _list/indices/index1,index2,index3?v&next_token=token
```
{% include copy-curl.html %}


## Example response

**Plain text format**

```json
health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size
green | open | movies | UZbpfERBQ1-3GSH2bnM3sg | 1 | 1 | 1 | 0 | 7.7kb | 3.8kb
next_token MTcyOTE5NTQ5NjM5N3wub3BlbnNlYXJjaC1zYXAtbG9nLXR5cGVzLWNvbmZpZw==
```

**JSON format**

```json
{"next_token":"MTcyOTE5NTQ5NjM5N3wub3BlbnNlYXJjaC1zYXAtbG9nLXR5cGVzLWNvbmZpZw==","indices":[{"health":"green","status":"open","index":"movies","uuid":"UZbpfERBQ1-3GSH2bnM3sg","pri":"1","rep":"1","docs.count":"1","docs.deleted":"0","store.size":"7.7kb","pri.store.size":"3.8kb"}]}
```
Loading

0 comments on commit ed84563

Please sign in to comment.