Identify data streams created by Ingest Manager with metadata

[cjcenizal]

In the Data Streams Management app, we'd like to identify data streams that were created via Ingest Management and provide a link back to a relevant location inside the Ingest Management app (per discussion with @mostlyjason).

Per discussion with @leehinman, @martijnvg, and @danhermann, we can update the Data Stream APIs with metadata support which will support this UX.

@mostlyjason Could you please clarify a useful place to link back to within Ingest Manager, so we can figure out what kind of information to store in the metadata?

Could the Ingest Manager engineers please coordinate with the ES API engineers to request the changes necessary in the Data Stream APIs?

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[danhermann]

According to elastic/elasticsearch#58316, we're planning to do this in a separate "data streams stats API" so that use cases that need only a basic list of data streams do not unnecessarily incur the overhead of retrieving all those stats. Please comment on that issue if there are any stats not listed there that are desired for the UI.

EDIT: Not relevant to this issue.

[cjcenizal]

Thanks @danhermann. To clarify, this issue is a request for something similar to the index template _meta property, so we can store arbitrary meta information, with the intention of using this property to tie a data stream back to a relevant entity within Ingest Manager. Would we still implement this in the stats API, or would this type of information live with the data stream itself?

[danhermann]

@cjcenizal, ah, thanks. I completely misunderstood. This meta information should be made available through the current get data streams API.

[ruflin]

@cjcenizal I think the linking is tricky at the moment as we don't have data stream details yet. So I would focus for now on just adding the meta information.

@danhermann We can add today _meta information to the template in 2 places: Index metadata and template metadata. I somehow assumed that the index metadata would be automatically the one for the data stream but it probably will make sense to separate these. Will there be a data stream specific section in the template for this?

[mostlyjason]

I created a separate issue for the metadata.

Also, we could deep link to the data streams page in ingest manager with a param to filter for the desired datastream. That might work even before we have detail pages.

[mostlyjason]

@cjcenizal we added this metadata to the index templates #70319. Do we also need it on the datastream or can you infer the relationship from the template?

[cjcenizal]

@mostlyjason That's great! I believe @sebelga will surface this information in our Index Templates UI. In order for me to surface something similar in the Data Streams UI, I will need this information to be applied to the data stream as well. There's no innate association between a data stream and the index template that configured it. Can you update the index templates to apply this metadata to the data streams they create?

[mostlyjason]

@cjcenizal our developer for this feature @neptunian said

For datastreams, I don't think we could this on our end as a data stream is not created until data is sent that matches an index template that has the data_stream property set.

Do you know of any way for the template to apply this metadata as you mentioned?

[cjcenizal]

@mostlyjason I just pinged the ES engineers and got confirmation that this isn't possible yet. It's something we'll get to, but not for 7.9. I created elastic/elasticsearch#59195 to track this enhancement.

[mostlyjason]

Adding a link to some UI mockups #64777

[jen-huang]

I see that Fleet data streams are showing up flagged in Index Management now:

So I'm going to close this issue, #79849 is still outstanding to provide a link back to Fleet somewhere, but we're still not sure on our side what a great place is, so that can still be left open for now.