GET data stream API returns additional information #59128

andreidan · 2020-07-07T09:51:42Z

This adds the data stream's index template, the configured ILM policy
(if any) and the health status of the data stream to the GET _data_stream
response.

Relates to #58316
Relates to #53100

The proposed response looks like this:

{
  "data_streams" : [
    {
      "name" : "timeseries",
      "timestamp_field" : {
        "name" : "@timestamp",
        "mapping" : {
          "type" : "date"
        }
      },
      "indices" : [
        {
          "index_name" : ".ds-timeseries-000001",
          "index_uuid" : "m3Id6-T7TuOBuC3qoltdqQ"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "timeseries_template",
      "ilm_policy" : "timeseries_policy"
    },
    {
      "name" : "timeseries_3",
      "timestamp_field" : {
        "name" : "@timestamp",
        "mapping" : {
          "type" : "date"
        }
      },
      "indices" : [
        {
          "index_name" : ".ds-timeseries_3-000001",
          "index_uuid" : "YJKrb9_yR2iukhi0vCy1zQ"
        }
      ],
      "generation" : 1,
      "status" : "GREEN",
      "template" : "timeseries_template",
      "ilm_policy" : "timeseries_policy"
    }
  ]
}

This adds the data stream's index template, the configured ILM policy (if any) and the health status of the data stream to the GET _data_stream response.

elasticmachine · 2020-07-07T09:51:44Z

Pinging @elastic/es-core-features (:Core/Features/Data streams)

andreidan · 2020-07-07T10:32:45Z

@elasticmachine update branch

andreidan · 2020-07-07T11:15:01Z

Added WIP as the HLRC still needs some work

danhermann

I left one comment about REST tests that I think should be addressed. LGTM, otherwise.

Edit: Just saw you marked it WIP. I'll take another look at it when you've completed it.

danhermann · 2020-07-07T12:16:41Z

rest-api-spec/src/main/resources/rest-api-spec/test/indices.data_stream/10_basic.yml

+  - match: { data_streams.0.name: simple-data-stream1 }
+  - match: { data_streams.0.timestamp_field.name: '@timestamp' }
+  - match: { data_streams.0.generation: 1 }
+  - length: { data_streams.0.indices: 1 }
+  - match: { data_streams.0.indices.0.index_name: '.ds-simple-data-stream1-000001' }
+  - match: { data_streams.1.name: simple-data-stream2 }
+  - match: { data_streams.1.timestamp_field.name: '@timestamp2' }
+  - match: { data_streams.0.generation: 1 }
+  - length: { data_streams.1.indices: 1 }
+  - match: { data_streams.1.indices.0.index_name: '.ds-simple-data-stream2-000001' }


I think that at least one of these REST tests should be updated with the full list of fields that are returned from the get data stream API.

Thanks Dan, pushed a match for status and template (the lifecycle policy is tested in TimeseriesDataStreamIT as it's an xpack setting)

danhermann

LGTM. Thanks, @andreidan!

Restoring a data stream from a snapshot could install a data stream that doesn't match any composable templates. This makes the index template field in the `GET _data_stream` response optional.

This adds the data stream's index template, the configured ILM policy (if any) and the health status of the data stream to the GET _data_stream response. Restoring a data stream from a snapshot could install a data stream that doesn't match any composable templates. This also makes the `template` field in the `GET _data_stream` response optional. (cherry picked from commit 0d9c98a) Signed-off-by: Andrei Dan <[email protected]>

martijnvg · 2020-07-07T17:53:20Z

server/src/main/java/org/elasticsearch/action/admin/indices/datastream/GetDataStreamAction.java

@@ -119,11 +214,13 @@ public void writeTo(StreamOutput out) throws IOException {

        @Override
        public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
-            builder.startArray();
-            for (DataStream dataStream : dataStreams) {
+            builder.startObject();


why was this top level data_streams field introduced? I don't think that this is needed?

It did make it easier to verify in REST tests that the number of returned data streams was as expected with a - length: { data_streams: 0 } directive. I don't think the length directive otherwise provides a way to refer to the root level of a response.

I think the - length: directive should be improved to handle this? Now the response format is changed, because a test directive can't handle something.

I also don't see length directive being used on the data_streams object, so maybe just keep using:

- match: { $body: [] }

?

Those are fair points. 👍

Went of the top array field to be consistent with component and composable templates eg.

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/indices/template/get/GetComponentTemplateAction.java

Line 146 in 3515909

builder.startArray(COMPONENT_TEMPLATES.getPreferredName());

…9177) * GET data stream API returns additional information (#59128) This adds the data stream's index template, the configured ILM policy (if any) and the health status of the data stream to the GET _data_stream response. Restoring a data stream from a snapshot could install a data stream that doesn't match any composable templates. This also makes the `template` field in the `GET _data_stream` response optional. (cherry picked from commit 0d9c98a) Signed-off-by: Andrei Dan <[email protected]>

Updates docs and snippets for changes made to the get data stream API with PR #59128.

GET data stream API returns additional information

e92adff

This adds the data stream's index template, the configured ILM policy (if any) and the health status of the data stream to the GET _data_stream response.

andreidan added v8.0.0 :Data Management/Data streams Data streams and their lifecycles Team:Deployment Management Meta label for Management Experience - Deployment Management team v7.9.0 labels Jul 7, 2020

elasticmachine added the Team:Data Management Meta label for data/management team label Jul 7, 2020

andreidan requested a review from martijnvg July 7, 2020 09:51

andreidan added 2 commits July 7, 2020 11:11

Remove unused import

4a4a356

Fix yaml tests

6c553b3

elasticmachine and others added 2 commits July 7, 2020 04:32

Merge branch 'master' into get_data_stream

57a8298

Fix yaml test

7dabe6e

andreidan removed the request for review from martijnvg July 7, 2020 11:14

andreidan added the WIP label Jul 7, 2020

danhermann approved these changes Jul 7, 2020

View reviewed changes

andreidan added 3 commits July 7, 2020 13:45

Adjust HLRC to new API

1a2d43d

Parse ilm_policy as optional

8340a54

Yaml test: match corresponding DS status and template

3fb57b1

andreidan removed the WIP label Jul 7, 2020

andreidan requested a review from danhermann July 7, 2020 13:30

danhermann approved these changes Jul 7, 2020

View reviewed changes

andreidan added 2 commits July 7, 2020 15:15

Fix stack yaml tests

af0feb6

Make the index template optional for the GET _data_stream response

4e66c0e

Restoring a data stream from a snapshot could install a data stream that doesn't match any composable templates. This makes the index template field in the `GET _data_stream` response optional.

andreidan merged commit 0d9c98a into elastic:master Jul 7, 2020

andreidan mentioned this pull request Jul 7, 2020

[7.9] GET data stream API returns additional information (#59128) #59177

Merged

martijnvg reviewed Jul 7, 2020

View reviewed changes

jrodewig mentioned this pull request Jul 7, 2020

[DOCS] Update get data stream API response #59197

Merged

jrodewig added a commit that referenced this pull request Jul 8, 2020

[DOCS] Update get data stream API response (#59197)

db38528

Updates docs and snippets for changes made to the get data stream API with PR #59128.

jrodewig added a commit that referenced this pull request Jul 8, 2020

[DOCS] Update get data stream API response (#59197) (#59230)

7bd93de

Updates docs and snippets for changes made to the get data stream API with PR #59128.

jrodewig added a commit that referenced this pull request Jul 8, 2020

[7.x] [DOCS] Update get data stream API response (#59197) (#59221)

d2c5a4c

Updates docs and snippets for changes made to the get data stream API with PR #59128.

martijnvg mentioned this pull request Jul 14, 2020

Expose more information and stats about a data stream #58316

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GET data stream API returns additional information #59128

GET data stream API returns additional information #59128

andreidan commented Jul 7, 2020 •

edited

Loading

elasticmachine commented Jul 7, 2020

andreidan commented Jul 7, 2020

andreidan commented Jul 7, 2020

danhermann left a comment •

edited

Loading

danhermann Jul 7, 2020

andreidan Jul 7, 2020

danhermann left a comment

martijnvg Jul 7, 2020

danhermann Jul 7, 2020

martijnvg Jul 7, 2020

danhermann Jul 7, 2020

andreidan Jul 8, 2020

GET data stream API returns additional information #59128

GET data stream API returns additional information #59128

Conversation

andreidan commented Jul 7, 2020 • edited Loading

elasticmachine commented Jul 7, 2020

andreidan commented Jul 7, 2020

andreidan commented Jul 7, 2020

danhermann left a comment • edited Loading

Choose a reason for hiding this comment

danhermann Jul 7, 2020

Choose a reason for hiding this comment

andreidan Jul 7, 2020

Choose a reason for hiding this comment

danhermann left a comment

Choose a reason for hiding this comment

martijnvg Jul 7, 2020

Choose a reason for hiding this comment

danhermann Jul 7, 2020

Choose a reason for hiding this comment

martijnvg Jul 7, 2020

Choose a reason for hiding this comment

danhermann Jul 7, 2020

Choose a reason for hiding this comment

andreidan Jul 8, 2020

Choose a reason for hiding this comment

andreidan commented Jul 7, 2020 •

edited

Loading

danhermann left a comment •

edited

Loading