Added hidden filter to data streams tab #85028

yuliacech · 2020-12-04T15:00:12Z

Summary

This PR adds support for hidden data streams in Index Management app. A new view button allows user to toggle several filters for data streams. Currently, there are 'Fleet-managed data streams' and 'hidden data streams'. Fleet-managed data streams are enabled by default, hidden data streams are disabled by default.
The code reuses view filter button created for index template list.

Screenshots

There are some data streams, but all are filtered out: table is displayed with no rows.

Hidden data streams with a label

When there are no data streams at all, the empty prompt is shown. This UI has not changed

How to test

A hidden data stream is usually already preexisting ilm-history-5. To create more hidden data streams:

PUT _index_template/test
{
  "template": {},
  "index_patterns": [
    "test"
  ],
  "data_stream": {
    "hidden": true   
  }
}

POST test/_doc
{
  "@timestamp": "2020-01-27"
}

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Release note

In Index Management app, hidden data streams are now displayed and indicated by a label.

yuliacech · 2020-12-07T10:29:41Z

@elasticmachine merge upstream

elasticmachine · 2020-12-09T11:17:50Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

alisonelizabeth

Nice work @yuliacech! This is looking great. I left a few comments in the code. I think we might be able to make the api integration tests a little more robust; let me know what you think.

I'd also like to test the deep-linking before I approve. Can you remind me on how/where to test that? I checked the index templates UI, but from what I could see we are only indicating if there is an associated data stream and not actually linking to the data stream.

I also had a few UX questions/suggestions (some of which might be better suited for a separate PR):

I have trouble understanding the difference between managed, system and hidden 😄. Should hidden actually be managed, or is it a different behavior for data streams? For example, the ILM index template is actually marked as managed in the UI, but the associated data stream is hidden.
Could we also add the badge to the details panel of the data stream? You can check out index templates as an example.

For system templates, we add an additional warning when deleting. Would this be applicable to hidden data streams?

alisonelizabeth · 2020-12-09T14:39:12Z

x-pack/plugins/index_management/public/application/lib/data_streams.tsx

@@ -6,10 +6,34 @@

 import { DataStream } from '../../../common';

-export const isManagedByIngestManager = (dataStream: DataStream): boolean => {
+export const isFleetManaged = (dataStream: DataStream): boolean => {
+  // TODO check if the wording will change to 'fleet'


Were you planning on addressing this TODO as part of this PR?

I'll remove this as soon as I get info from the Fleet team :)

alisonelizabeth · 2020-12-09T14:53:38Z

...t/public/application/sections/home/data_stream_list/data_stream_table/data_stream_badges.tsx

+  dataStream: DataStream;
+}
+
+export const DataStreamsBadges: React.FunctionComponent<Props> = ({ dataStream }) => {


Is it possible that a fleet-managed data stream could also be hidden? Are there any UX concerns with displaying two badges side-by-side in a table row?

alisonelizabeth · 2020-12-09T14:54:33Z

x-pack/plugins/index_management/server/routes/api/data_streams/register_get_route.ts

@@ -65,12 +65,24 @@ const enhanceDataStreams = ({
  });
 };

+const getDataStreams = (client: ElasticsearchClient, name = '') => {
+  // TODO update when elasticsearch client has update requestParams for 'indices.getDataStream'


Is there an open issue that we can point to as part of this comment?

There is no issue as elasticsearch-js client generates their typings for API automatically. I think we just need to wait for the next client update, so I left this comment. But let me know what you think :)

alisonelizabeth · 2020-12-09T14:57:58Z

x-pack/plugins/index_management/server/routes/api/data_streams/register_get_route.ts

@@ -65,12 +65,24 @@ const enhanceDataStreams = ({
  });
 };

+const getDataStreams = (client: ElasticsearchClient, name = '') => {


Should we just not include name in the path if it isn't defined?

I think the code would not be much easier with a ternary operator for path, so I'd prefer to have a default value. Or did you mean something else?

My main concern is the path ends up resolving to /_data_stream// is there is no name specified

Thanks for bringing this up, @alisonelizabeth ! I haven't thought about it and that's a valid concern. I changed the empty string to a wildcard * and deleted the closing forward slash. So if name isn't defined, the path would be /_data_stream/*.

alisonelizabeth · 2020-12-09T15:00:55Z

x-pack/test/api_integration/apis/management/index_management/data_streams.ts

@@ -61,20 +61,44 @@ export default function ({ getService }: FtrProviderContext) {

  describe('Data streams', function () {
    describe('Get', () => {
+      const defaultHiddenDataStream = {


This seems a little brittle to provide values for the data stream generated by ES. For example, are we confident that the name will always beilm-history-5 ?

alisonelizabeth · 2020-12-09T15:12:21Z

x-pack/test/api_integration/apis/management/index_management/data_streams.ts

@@ -101,11 +126,13 @@ export default function ({ getService }: FtrProviderContext) {
          .expect(200);

        // ES determines these values so we'll just echo them back.
-        const { name: indexName, uuid } = dataStreams[0].indices[0];
-        const { storageSize, ...dataStreamWithoutStorageSize } = dataStreams[0];
+        // dataStreams[0] is a hidden data stream created by ES


I wonder if the test would be more resilient if we used find to find the datastream created in the test. I have concerns about the current implementation if ES were to add more hidden data streams. WDYT?

yuliacech · 2020-12-10T13:35:39Z

@elasticmachine merge upstream

yuliacech · 2020-12-10T18:27:10Z

Hi @alisonelizabeth , thank you so much for your review! I addressed the feedback, would you mind having another look?

I think 'managed' data streams will usually not be 'hidden', but if that happens I put the badges into a badge group, so that they would stack on a smaller screen. WDYT?

Added the badges to the detail flyout as well

As far as I understand, hidden data stream can be deleted, because this is just a property of a data stream set via an index template. I would not add any special information there, as there is currently no semantics attached to this property.
I see how that can be confusing, but managed templates are not connected to managed data streams. Those are data streams managed by Fleet. Do you think we should change 'managed' label to 'Fleet-managed' or similar?
Thanks for your notes about the api integration tests, I updated them to be more resilient against ES changes.
To check cross-linking, you could open Indices list, enable hidden indices and click on data stream name 'ilm-history-5'. I also added a query param, should we need to cross-link to data streams tab with hidden filter on with includeHidden=true.

alisonelizabeth

Thanks for addressing my feedback @yuliacech! I think the CITs need to be updated to react to your most recent changes, but otherwise LGTM. Tested again and verified the cross-linking.

Also, the badge on the details panel is misaligned for me - possibly caused by using the EuiBadgeGroup?

alisonelizabeth · 2020-12-10T19:06:06Z

x-pack/plugins/index_management/server/routes/api/data_streams/register_get_route.ts

@@ -65,12 +65,24 @@ const enhanceDataStreams = ({
  });
 };

+const getDataStreams = (client: ElasticsearchClient, name = '') => {


My main concern is the path ends up resolving to /_data_stream// is there is no name specified

alisonelizabeth · 2020-12-10T19:26:43Z

I think 'managed' data streams will usually not be 'hidden', but if that happens I put the badges into a badge group, so that they would stack on a smaller screen. WDYT?

👍

I see how that can be confusing, but managed templates are not connected to managed data streams. Those are data streams managed by Fleet. Do you think we should change 'managed' label to 'Fleet-managed' or similar?

I'm not as familiar with this area, but if they are always managed by Fleet I think it would be helpful to update the label to be more specific. As a future enhancement, I think it could also be helpful to provide a tooltip or something to each badge that gives a better explanation of what it means.

yuliacech · 2020-12-14T11:22:40Z

@elasticmachine merge upstream

yuliacech · 2020-12-14T13:31:48Z

Thanks for re-reviewing @alisonelizabeth ! I fixed the alignment issue in the detail panel :) Will create a separate PR for wording of Fleet-managed data streams and a possible tooltip for the label.

kibanamachine · 2020-12-14T15:43:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 01eb4ff

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`indexManagement`	523	525	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexManagement`	1.5MB	1.5MB	+2.0KB

Distributable file count

id	before	after	diff
`default`	47129	47889	+760

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`indexManagement`	124.1KB	124.1KB	+45.0B

History

💔 Build #93948 failed 865b781
💔 Build #93611 failed 8d13d44
💚 Build #93475 succeeded 7475605
💚 Build #92396 succeeded 6629910
💔 Build #92290 failed ea7380c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Added hidden filter to data streams tab * Fix i18n import * Fixed tests * hidden ds pr feedback * Added includeHidden query to data streams list * Changed how badge group renders data streams badges Co-authored-by: Kibana Machine <[email protected]>

* Added hidden filter to data streams tab * Fix i18n import * Fixed tests * hidden ds pr feedback * Added includeHidden query to data streams list * Changed how badge group renders data streams badges Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

yuliacech added 2 commits December 4, 2020 15:31

Added hidden filter to data streams tab

9afa24b

Fix i18n import

3cd3321

kibanamachine and others added 2 commits December 7, 2020 05:29

Merge branch 'master' into hidden_ds

ea7380c

Fixed tests

6629910

yuliacech added Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.11.0 v8.0.0 release_note:enhancement labels Dec 9, 2020

yuliacech marked this pull request as ready for review December 9, 2020 11:17

yuliacech requested a review from a team as a code owner December 9, 2020 11:17

yuliacech requested a review from alisonelizabeth December 9, 2020 11:19

alisonelizabeth reviewed Dec 9, 2020

View reviewed changes

kibanamachine and others added 3 commits December 10, 2020 08:35

Merge branch 'master' into hidden_ds

7475605

hidden ds pr feedback

04b2f2f

Added includeHidden query to data streams list

8d13d44

alisonelizabeth approved these changes Dec 10, 2020

View reviewed changes

kibanamachine and others added 2 commits December 14, 2020 06:22

Merge branch 'master' into hidden_ds

865b781

Changed how badge group renders data streams badges

01eb4ff

yuliacech merged commit a8d088f into elastic:master Dec 14, 2020

yuliacech mentioned this pull request Dec 14, 2020

[7.x] Added hidden filter to data streams tab (#85028) #85800

Merged

yuliacech mentioned this pull request Dec 14, 2020

[Index Management][Datastreams] Fixed datastreams functional app load test to check for visibility of element to prevent false positives #85708

Merged

sebelga mentioned this pull request Jan 5, 2021

[Index template] Data stream configuration is removed #87293

Closed

yuliacech mentioned this pull request Jan 7, 2021

[DOCS] Update data streams list screenshots and description of "delete data stream" elastic/elasticsearch#67145

Merged

yuliacech deleted the hidden_ds branch February 24, 2021 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added hidden filter to data streams tab #85028

Added hidden filter to data streams tab #85028

yuliacech commented Dec 4, 2020 •

edited

Loading

yuliacech commented Dec 7, 2020

elasticmachine commented Dec 9, 2020

alisonelizabeth left a comment

alisonelizabeth Dec 9, 2020

yuliacech Dec 10, 2020

alisonelizabeth Dec 9, 2020

alisonelizabeth Dec 9, 2020

yuliacech Dec 10, 2020

alisonelizabeth Dec 9, 2020

yuliacech Dec 10, 2020

alisonelizabeth Dec 10, 2020

yuliacech Dec 14, 2020

alisonelizabeth Dec 9, 2020

alisonelizabeth Dec 9, 2020

yuliacech commented Dec 10, 2020

yuliacech commented Dec 10, 2020

alisonelizabeth left a comment

alisonelizabeth Dec 10, 2020

alisonelizabeth commented Dec 10, 2020 •

edited

Loading

yuliacech commented Dec 14, 2020

yuliacech commented Dec 14, 2020

kibanamachine commented Dec 14, 2020

Added hidden filter to data streams tab #85028

Added hidden filter to data streams tab #85028

Conversation

yuliacech commented Dec 4, 2020 • edited Loading

Summary

Screenshots

How to test

Checklist

Release note

yuliacech commented Dec 7, 2020

elasticmachine commented Dec 9, 2020

alisonelizabeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuliacech commented Dec 10, 2020

yuliacech commented Dec 10, 2020

alisonelizabeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alisonelizabeth commented Dec 10, 2020 • edited Loading

yuliacech commented Dec 14, 2020

yuliacech commented Dec 14, 2020

kibanamachine commented Dec 14, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Distributable file count

Page load bundle

History

yuliacech commented Dec 4, 2020 •

edited

Loading

alisonelizabeth commented Dec 10, 2020 •

edited

Loading