Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In addition to the 409 errros, also ignore valid 201 created document… #8981

Conversation

dsteininger86
Copy link

…s within a "errors=true" elastic bulk response

It actually only covers this issue: #6341


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

…s within a "errors=true" elastic bulk response

Signed-off-by: Daniel Steininger <[email protected]>
@dsteininger86
Copy link
Author

ping @PettitWesley

* Check for errors other than version conflict (document already exists)
* and ignore the created documents.
*/
if (item_val.via.i64 != 409 && item_val.via.i64 != 201) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that this shouldn't be covered for the following case (error, succeeded, succeeded, error case):

{
 "took": 15022,
 "errors": true,
 "items": [
  {
   "create": {
    "_index": "platform-filebeat-2022.10.26-000671",
    "_type": "_doc",
    "_id": "f52b988e-ee72-79b4-7854-863f517883d4",
    "status": 429,
    "error": {
     "type": "es_rejected_execution_exception",
     "reason": "rejected execution of org.elasticsearch.action.support.replication.TransportWriteAction$1/WrappedActionListener{org.elasticsearch.action.support.replication.ReplicationOperation$$Lambda$6662/0x0000000801b00b08@52f3939d}{org.elasticsearch.action.support.replication.ReplicationOperation$$Lambda$6666/0x0000000801b013a8@1a3c5f59} on EsThreadPoolExecutor[name = elasticsearch-es-default-resized-0/write, queue capacity = 10000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@7675026a[Running, pool size = 4, active threads = 4, queued tasks = 10134, completed tasks = 10934096]]"
    }
   }
  },
  {
   "create": {
    "_index": "platform-filebeat-2022.10.26-000671",
    "_type": "_doc",
    "_id": "5cb12418-c53d-bcc0-4d8e-25ca508c238c",
    "_version": 1,
    "result": "created",
    "_shards": {
     "total": 2,
     "successful": 2,
     "failed": 0
    },
    "_seq_no": 801595,
    "_primary_term": 1,
    "status": 201
   }
  },
  {
   "create": {
    "_index": "platform-filebeat-2022.10.26-000671",
    "_type": "_doc",
    "_id": "196d923e-0e07-10f0-b091-ae5cf142a7a7",
    "_version": 1,
    "result": "created",
    "_shards": {
     "total": 2,
     "successful": 2,
     "failed": 0
    },
    "_seq_no": 806215,
    "_primary_term": 1,
    "status": 201
   }
  },
  {
   "create": {
    "_index": "platform-filebeat-2022.10.26-000671",
    "_type": "_doc",
    "_id": "5cb12418-c53d-bcc0-4d8e-25ca508c238c",
    "status": 409,
    "error": {
     "type": "version_conflict_engine_exception",
     "reason": "[5cb12418-c53d-bcc0-4d8e-25ca508c238c]: version conflict, document already exists (current version [1])",
     "index_uuid": "x9HHbHGpRdiwMjDFgvdETQ",
     "shard": "4",
     "index": "platform-filebeat-2022.10.26-000671"
    }
   }
  }
 ]
}

This is because check variable should be replaced by the last failure result.
Am I missing some of the conditions?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented another fix here: #9236

@dsteininger86
Copy link
Author

dsteininger86 commented Sep 5, 2024

closed, as the problem seems to be fixed in the following fluent-bit:v3.1.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants