Infinite ingestion retry when batches are too large and using GuaranteedSend

[simitt]

Elasticsearch returns status code 413 when a bulk request exceeds the size limit. A user can either increase the http.max_content_length in ES or decrease the bulk_max_size in the Beat to overcome such failures.
However, when this error happens and the beat is using a GuaranteedSend publisher method the current implementation can lead to an infinite retry, sending the same request to ES.
This might result in not being able to ingest any more events.

It might be worth exploring to use a special handling for the batch when the request size exceeds a limit, e.g. split it in half.

[ph]

Related issue #3688

[ph]

Prior experience with that from LS logstash-plugins/logstash-output-elasticsearch#497

[ph]

Linked to #6749

[faec]

This issue probably still exists, but seems rare, is fixable with proper configuration, and was never allocated time in a release cycle -- unassigning so it can be re-triaged.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

Ping @mukeshelastic @nimarezainia as you were both interested by this issue. It will be fixed to 8.1 thanks To @rdner

[mukeshelastic]

Thanks @jlind23

[dikshachauhan-qasource]

Hi @simitt

Could you please help us on this Ticket validation with below points:

• How can we create bulk requests for elasticsearch?
• Can it be covered under manual testing.

Thanks
QAS

[simitt]

@rdner given that you implemented the fix, can you please provide guidance for the testers.
It's been a long time since I created the issue, and don't believe this is reproducible anymore with the latest apm-server version, as it is not using libbeat output to ES anymore.

[rdner]

@dikshachauhan-qasource I described the testing process in my PR #29368

Let me know if it's missing something.