Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Uses ES bulk api only when there's more than one span
During a test where 100 single-span messages are sent to Kafka at the same time, I noticed only 58-97 of them would end up in storage eventhough all messages parsed properly and no operations failed. After a 100ms/message pause was added, the store rate of this test went to 100%, so figured it was some sort of state issue. I noticed the code was using Bulk operations regardless of input size, so as a wild guess changed the special-case single-span messages. At least in this test, it raised the success rate to 100% without any pausing needed. I don't know why this worked, but it seems sensible to not use bulk apis when there's no bulk action to perform. I started to write a unit test to validate single-length lists don't use bulk, but the Mockito involved became too verbose as the Elasticsearch client uses chaining and other patterns that are tedious to mock. Instead, we should make a parallel integration test and apply them to all storage components. See #1141
- Loading branch information