Add the write_operation and upsert mode of the ES Fluentd plugin

[obilliard]

Is your feature request related to a problem? Please describe.
We are migrating from Logstash to Fluent Bit.
Our system is generating a log when a query start and when a query ends. The logs are pushed to ElasticSearch. The logs are processed asynchronously so we cannot ensure the order in which they are handled by Logstash. We are generating our own ES document_id and writing to ES using the update action and an upsert script. This way, we end up with an ES entry holding 2 timestamps. We monitor the status of the running queries (entries in ES having a start timestamp but no end timestamp) in a Grafana dashboard.
Currently the ES output plugin only provide an insert action.

Describe the solution you'd like
I would like Fluent Bit handling the update and upsert mode.

Describe alternatives you've considered
The Fluentd ES plugin allows to choose the write operation, which could be either update or upsert.

Additional context
This is a current limitation that removes a current feature provided by our system.

[rigsby-chi]

That's actually a great idea. I wanted to do a similar setup before but failed because fluent-bit does not support upsert mode. So ends up I am still using Filebeat+Logstash.

[edsiper]

Since I am not familiar with all the details of Elasticsearch; from a Fluent Bit perspective, this means you would like an option kind of "Upsert True" and then the request uses the Upsert header in the payload ?

[rigsby-chi]

Yep. Also, it should allow the user to specify an existing field to be used as the _id (instead of generating by FluentBit or Elasticsearch). Applications write multiple logs containing the same ID, then FluentBit uses this ID field as the _id and performs upsert operation. So eventually, all these logs will be in the same document in Elasticsearch.

e.g.
An application generates two logs for the start and end of an operation (may be a query exeuction)
{"exe-id": "a71rq", "startTime": "11:12", "query": "xxxxx"}
{"exe-id": "a71rq", "endTime": "11:30", "result":"success"}

After going through FluentBit with upsert mode, they come one single document in elasticsearch
{"_id": "a71rq", "exe-id": "a71rq", "startTime": "11:12", "endTime": "11:30", "query": "xxxxx", "result":"success"}

By searching for documents that have startTime without endTime, administrators can easily know which queries are still running.

[obilliard]

Hi @edsiper,
More information has been provided.
Do you have any update on this?

[trexx]

This could still be useful in order to take advantage of ES 7.9 Data Streams which only support the create write_operation.