Lazy data stream rollover is not triggered when using reroute

[axw]

Elasticsearch Version

8.15.1

Installed Plugins

No response

Java Version

bundled

OS Version

N/A

Problem Description

Lazy rollover on a data stream is not triggered when writing a document that is rerouted to another data stream. This affects the apm-data plugin, where we perform a lazy rollover of matching data stream patterns when installing or updating index templates. The data stream never rolls over. See elastic/apm-server#14060 (comment)

Should a write that leads to a reroute also trigger the lazy rollover? I think so, otherwise the default pipeline will not change.

Steps to Reproduce

1. Create an index template which sets a default ingest pipeline with reroute

PUT /_ingest/pipeline/demo-reroute
{
  "processors": [
    {
      "reroute": {"namespace": "foo"}
    }
  ]
}

PUT /_index_template/demo_1
{
  "index_patterns" : ["demo*"],
  "data_stream": {}, 
  "priority" : 1,
  "template": {
    "settings" : {
      "number_of_shards": 1,
      "index.default_pipeline": "demo-reroute"
    }
  }
}

2. Create a data stream matching the index template

PUT /_data_stream/demo-dataset-default

3. Send a document to the data stream; it will be rerouted

POST /demo-dataset-default/_doc
{
  "@timestamp": "2024-09-12"
}

{
  "_index": ".ds-demo-dataset-foo-2024.09.12-000001",
  "_id": "z2Ab5JEBCHevSrCVP7aG",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

4. Create another index template with higher priority with the same index pattern, with no default ingest pipeline

PUT /_index_template/demo_2
{
  "index_patterns" : ["demo*"],
  "data_stream": {}, 
  "priority" : 2
}

5. Rollover the source data stream with "lazy=true"

POST /demo-dataset-default/_rollover?lazy=true

6. Send a document to the data stream; it will still be rerouted

POST /demo-dataset-default/_doc
{
  "@timestamp": "2024-09-12"
}

{
  "_index": ".ds-demo-dataset-foo-2024.09.12-000001",
  "_id": "x2gc5JEBfAEizTaQVStE",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}

7. Rollover the source data stream with "lazy=false"

POST /demo-dataset-default/_rollover?lazy=false

8. Send a document to the data stream; it will not be rerouted

POST /demo-dataset-default/_doc
{
  "@timestamp": "2024-09-12"
}

{
  "_index": ".ds-demo-dataset-default-2024.09.12-000002",
  "_id": "1mAf5JEBCHevSrCVc7YV",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[axw]

Is this a more general case of lazy rollover only being triggered post ingest pipeline, and not specific to rerouting? We're also seeing issues related to upgrading from older versions of of APM (e.g. 8.12.1) to 8.15.1, without any reroute processor involved.

[gmarouli]

Hi @axw, we did not know you were doing version checks on the pipelines, so yes, that is definitely a side effect of the lazy rollover happening only upon a write to the index. The timing of the rollover is important though because if we rollover earlier we risk creating empty indices.

We discussed possible approaches to solve this in a way that does not produce extra indices and we have the following proposal:

1. When a data stream is marked for a lazy rollover (and only then)
2. We we would resolve the template and retrieve the most up-to-date pipeline to be executed.

This way we have the following benefits:

• We are using the latest pipeline
• We do not rollover if not necessary (aka if there is a reroute processor)

The drawbacks:

• If the data stream marked for a lazy rollover has a pipeline with the reroute processor, we risk resolving the templates once per request "forever" since no writes will reach that data streams. We should check the overhead this adds to the indexing process.

[axw]

@gmarouli thanks, sounds reasonable. Just to clarify, we don't do version checks in recent versions of our ingest pipeline - that only applies to versions before 8.13.0.

If the data stream marked for a lazy rollover has a pipeline with the reroute processor, we risk resolving the templates once per request "forever" since no writes will reach that data streams. We should check the overhead this adds to the indexing process.

+1 that was also my first thought.

Would it make sense to extend this approach to also update the marked data stream after executing the pipeline if there were no writes?

[gmarouli]

Would it make sense to extend this approach to also update the marked data stream after executing the pipeline if there were no writes?

What do you mean with this?

[axw]

@gmarouli sorry, that was very unclear, let me try again.

If the data stream is marked for lazy rollover, do what you described where we resolve any settings (e.g. ingest pipeline) that may affect ingestion from the matching index template; then if there was a change in template, execute the rollover even if there were no writes to the data stream's backing index. That way we wouldn't need to do the template resolution on every write to the data stream, only once per lazy rollover.

[gmarouli]

@axw thank you for the explanation, I get it now.

You are right, that would address the potential latency but we would be creating empty indices which is something we want to avoid. Let's say what's the impact and if it can be sustained until we have a more structural solution available.