Add Managed label to data streams and a view switch for the table

[yuliacech]

Summary

This PR adds a 'Managed' label to data streams managed by Ingest Manager. There is also a new 'Include managed' switch, that is enabled by default. When disabled, managed data streams are hidden.

Screenshot

gif

How to test

1. start Kibana with yarn start --xpack.fleet.enabled=true --no-base-path
2. start ES with yarn es snapshot -E xpack.security.authc.api_key.enabled=true
3. Download an Elastic agent for your OS from here and unpack the archive to a folder
4. In Kibana go to Fleet -> Agents and click the button 'Create user and enable central management'
5. After that click 'Add agent' button and in the flyout look for the enrolment token in section 3. (but don't copy the command, it doesn't work)
6. In the folder where you unpacked downloaded Elastic agent, run ./elastic-agent enroll http://localhost:5601 $ENROLMENT_TOKEN --insecure
7. In the folder where you unpacked downloaded Elastic agent, run ./elastic-agent run and keep this terminal tab open.
8. You should see data streams in Fleet and in Index Management.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

Release Note

In the Index Management app, data streams managed by Fleet can now be easily identified by a 'Managed' label.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[yuliacech]

@elasticmachine merge upstream

[alisonelizabeth]

Great work @yuliacech! TIL how to set up Ingest Manager 😄.

Tested locally and everything worked as expected. I left a few comments, but nothing blocking.

[alisonelizabeth]

This is a little hard for me to parse. If I'm understanding correctly, it looks like managed is only ever going to be false.

const filteredDataStreams = isIncludeManagedChecked
      ? dataStreams
      : filterDataStreams(dataStreams, isIncludeManagedChecked);

If that's correct, could the filter be simplified to this?

return dataStreams.filter((dataStream: DataStream) => isManagedByIngestManager(dataStream) === false)

[alisonelizabeth]

(nit) you can use shorthand <></> for fragments.

[alisonelizabeth]

WDYT about changing this to "Include managed" to align with the other toggle?

[alisonelizabeth]

What do you think about adding a tooltip or title to the badge to explain that "managed" means managed by Ingest Manager, specifically? Or maybe add this to the switch?

[yuliacech]

@elasticmachine merge upstream

[yuliacech]

Thank you for the review, @alisonelizabeth ! I renamed the switch and added some tooltips as you suggested :)

[yuliacech]

Hello @esdocs , could you please have a look at this PR's wording? I would appreciate some feedback on tooltips I added to the "Managed" label and to the "Include managed" switch (see screenshots in PR description). I'm not sure if I should use "Ingest Manager" or "Fleet" since the plugin name's change.

[jrodewig]

Thanks for reaching out about this PR @yuliacech. At the least, I think we'll need to replace Ingest Manager with Fleet throughout. However, I also think the current copy places too much emphasis on "manage" and may confuse ILM users.

What we're really trying to communicate here is that these are internal system data streams that the user generally doesn't need to view or touch. We may not even need to point out that they're managed by Fleet.

EDIT: I was mistaken about the purpose of these managed streams. I'll submit another review.

I left some related suggestions for the copy. I'd also like to get @dedemorton's approval before merging. She's been writing the Fleet docs and is most familiar with the current terms.

[cjcenizal]

@jrodewig I'm a little confused by this:

What we're really trying to communicate here is that these are internal system data streams that the user generally doesn't need to view or touch. We may not even need to point out that they're managed by Fleet.

My understanding of the term "system" is informed by my understanding of our original dot-prefixed system indices, like .kibana, .watches, and .security, which are being replaced by the formalized Elasticsearch system indices. These indices match your description of "internal" data that the user shouldn't be concerned with. Our Stack is responsible for managing them, not the user.

However, my understanding of Fleet data streams is that the user is concerned with them. These are created by the user when the user adds a new integration via Fleet, they store data ingested by the user's Elastic Agent, and the user will depend on visualizations and dashboards that consume this data. If the user wants to finesse storage costs, they'll fiddle with SLM and ILM to back up this data and move it to cheaper storage. Given this definition of Fleet's data streams, it seems useful to differentiate them from true system indices (and system data streams, once those exist).

So I'm sensing a gap between the way we're seeing things, and I'm wondering if I'm mistaken in my understanding of things. Can you help me understand your perspective so we can bridge the gap?

[jrodewig]

Hey @cjcenizal! Thanks for commenting. I'm likely wrong here. My main confusion arose from the -default streams vs streams like metric-system. But peaking at the data in those streams, you seem to be right. I'm a relative newbie to Elastic Agent, so my fault for not reaching out before submitting a review. Apologies.

I'll resolve my suggestions and submit a new review. However, I'll leave my original comments up for context. Thanks again!

[jrodewig]

LGTM. Apologies for my earlier misunderstanding. I left some terminology-related suggestions, but I'd still like approval from @dedemorton before merging. Thanks!

[yuliacech]

@elasticmachine merge upstream

[dedemorton]

Text strings LGTM.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 69f4146

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
indexManagement	511	512	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
indexManagement	1.5MB	1.5MB	+2.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
indexManagement	114.1KB	114.2KB	+41.0B

History

• 💚 Build #88645 succeeded 230efd5
• 💚 Build #88646 succeeded 69f4146
• 💚 Build #88644 succeeded ea4c669
• 💚 Build #88522 succeeded 0c7efef
• 💚 Build #87988 succeeded 33469e4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream