Support for TSDB in package spec

[ruflin]

Elasticsearch is working on the time series data based (TSDB) project: elastic/elasticsearch#74660 This introduces new mapping fields for metrics and also a time series id (TSID). A first version of this project will land soon and parts are already available in the recent snapshot builds. To make this feature available to integrations, these fields have to be added to the package-spec and be supported by Fleet to be installed.

Requirements:

• Support for dimensions.
• Add some setting to enable TSDB mode in the index.

[ruflin]

• @imotov Could you share a link to the docs and the fields that need to be added to the package spec?
• @joshdover For awareness as this likely needs changes in Fleet
• @jportner My assumption is that we can start using these fields and until Lens natively supports it, we can just keep using the old queries without getting all the benefits?

[joshdover]

Is this different than #297?

[joshdover]

We also already added support for the dimensions and metric mapping options in elastic/kibana#115620 and elastic/kibana#115621. These are shipping 8.2.0

[ruflin]

This is great! Looks like @jsoriano is way ahead of me and #236 + #297 cover already most (all) of it. Now we need some integrations making use of it for testing.

[jportner]

• @jportner My assumption is that we can start using these fields and until Lens natively supports it, we can just keep using the old queries without getting all the benefits?

Did you mean to ping the other Joe (@flash1293) instead of me? 😄

[flash1293]

That makes sense because this topic is very much related to what I’m doing 🙂 as tsdb indices can be queried as regular indices, you will be able to use all of the Lens functionality with them - in this way it will support them from the start. We are going to add new functionality leveraging the time_series and sliding window aggregations later on to Lens.

[jsoriano]

Closing this as at the end no changes are needed in the spec, we will use current Elasticsearch options:

elasticsearch:
  index_template:
    settings:
      index.mode: time_series

Enabling it this way in the kubernetes integration, here: elastic/integrations#3119.

[jsoriano]

Let's reopen this as public-facing issue, and because at the end we may need changes in the spec.