Enabling tsdb on a standard datastream that have been a tsdb datastream previously fail with overlapping backing index

[nchaulet]

Elasticsearch Version

8.12.2

Installed Plugins

No response

Java Version

bundled

OS Version

macos

Problem Description

Enabling TSDB on a datastream that has been a TSDB before but that was rollbacked to a standard stream, can cause backing index ... is overlapping with backing index ...

It's a Fleet use case when we fail to upgrade an integration we rollback to a previous version, the future upgrade are then failing. It seems it will work if we wait as there will be no overlapping index anymore. But some Fleet process can trigger automatic upgrades and having to wait 4 hours to be able to upgrade again is not optimal, both for Fleet and for our users.

In my opinion this probably should be automatically handled by Elasticsearch

cc @salvatore-campagna

Steps to Reproduce

Enabling TSDB on a datastream that has been a TSDB before but that was rollbacked to a standard stream, can cause backing index ... is overlapping with backing index ...

For example in fleet:

1. we install a package this create the datastream metrics-no_tsdb_to_tsdb.test-default without tsdb => this create a backing index 0001
2. We upgrade, update metrics-no_tsdb_to_tsdb.test-default datastream to TSDB => this create a timeseries backing index 0002
3. we rollback thepckage, update metrics-no_tsdb_to_tsdb.test- and rollover without tsb => this create backing indices 0003, 0004 without timeseries
4. We try to upgrade again metrics-no_tsdb_to_tsdb.test-default datastream to TSDB => this fail with

backing index [.ds-metrics-no_tsdb_to_tsdb.test-default-2024.09.20-000002] with range [2024-09-19T22:49:57.000Z TO 2024-09-20T01:19:57.000Z] is overlapping with backing index [.ds-metrics-no_tsdb_to_tsdb.test-default-2024.09.20-000005] with range [2024-09-19T22:50:04.000Z TO 2024-09-20T01:20:04.000Z]

Example with Fleet here elastic/kibana#193503

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

Duplicate of #96163

This is a known issue. Downgrading from tsdb and then immediately upgrading to tsdb fails now, but previous backing index doesn't have index.time_series.start_time and index.time_series.end_time settings. And the new backing index that is attempted to be created overlaps with the 3th recent backing index.

By default if the upgrade is executed 2 hours or later after the data stream was downgraded to non tsdb, then upgrading to tsdb will work. This depends on the index.look_back_time and index.look_ahead_time index settings in the template.