v1.8.2 not pushing logs to elasticsearch, giving action_request_validation_exception

[abhisheksharma2805]

{"create":{"_index":"fluent-bit-temp-2021.08.04","_type":"doc"}}
// Json document {}
[2021/08/04 08:42:58] [error] [output:es:es.0] HTTP status=400 URI=/_bulk, response:
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;"},"status":400}

[abhisheksharma2805]

Everything was working fine till v1.6.10. I wanted to use the new multiline filters and parsers so I shifted to v1.8.2.
I think the issue might be in this line

{"create":{"_index":"fluent-bit-temp-2021.08.04","_type":"doc"}}

[jlintz]

I believe this may be related to 7f0db9e#diff-6b9d334dc0b7304ad9aa29adc05402c652e1776b99f651ebf181e5547e3ab4e6 and changing of the bulk index format

[abhisheksharma2805]

@jlintz Thanks for pointing this out.
Turning Generate_ID option to On works now. This will generate unique doc id everytime. I believe this should be On by default.

        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}       
        Generate_ID     On      
        Logstash_Format On
        Logstash_Prefix fluent-bit-temp
        Retry_Limit     False
        Replace_Dots    On

[mut3]

Specifically this change 7f0db9e#diff-3fc4adb0ddceaceddc7686c08d19a9ea90c5cf3a79fb0e1c14a3d14bc64a6257L28-R31 probably caused it

I am sending to an elasticsearch 6.8 cluster and using Fluent Bit v1.7.9 worked, I had the same error on v1.8.x

[BertelBB]

I believe this may be related to 7f0db9e#diff-6b9d334dc0b7304ad9aa29adc05402c652e1776b99f651ebf181e5547e3ab4e6 and changing of the bulk index format

I suspect there is a bug in the code from reading that description. I have been noticing a lot of 409 status codes from elastic after upgrading to 1.8.x and they are logged as errors in fluent-bit.

We solve this by replacing all "index" operations by the "create"
operation. However, this has the side effect of producing status 409
errors whenever a previously successful operation is retried and the
Generate_ID option is turned on. Therefore, we change the
"elasticsearch_error_check" function to ignore this kind of error.

Also, why retry a previously successful operation?

[Dentrax]

kind ping @fujimotos @thotypous 😇

any updates here? Is this known bug or done intentionally?

As a workaround, we set the Generate_ID On in the values.yml (helm) (version: 1.8.6) as @abhisheksharma2805 said. 🙏

[thotypous]

It used to work with Generate_ID Off back when I initially submitted the patch, but that was a really long time ago. Maybe generating the ID is now required. Is there an use case for not generating the ID? Maybe it could be enforced as @abhisheksharma2805 suggested.

[Albelt]

It works with Generate_ID On, however the doc doesn't mention this point, which is unfriendly to new hand like me 🤣

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[Dentrax]

Sorry for the pingin. 🙏 @edsiper @fujimotos @thotypous @PettitWesley

Can someone please enlighten us about this problem and clarify the whether it's bug or intentional? I could not see a mention of this change in the v1.8.0 changelog. Are there any mitigation guide for the major releases?

Thanks in advance. ❤️

[PettitWesley]

@Dentrax Using Generate_ID solves this right?

If you feel the documentation needs to be improved, please open an issue here: https://github.com/fluent/fluent-bit-docs

[fujimotos]

If you feel the documentation needs to be improved, please open an issue here: https://github.com/fluent/fluent-bit-docs

I added that documentation.

As I recall, we needed this change to support Elasticsearch Datastream. This is (AFAIK) the recommended way nowadays to handle time-based data such as application logs.

This subtly changed the semantics about _id, which is now mandated because Elasticsearch requires
an explicit id for each record. So I don't think this is technically a bug, but just a requirement of Elasticsearch API (on which out_es is built).