High number of storage chunks slows down pipeline

[edsiper]

Bug Report

When the storage backlog system has thousands of pending chunks, the start of the service is slow and it might slow down the delivery of data.

This issue has been reported by @rmacian. Confirmed through Zoom video session.

[edsiper]

The root cause of the problem:

when thousands of data chunks exist, the Chunkio file handler needed to count how many chunks were up in memory, this operation was called many times generating high CPU and generating latency to the service.

The new version of Chunk I/O 1.0.1 added native counters as a state instead of count every time, no more performance issues are seeing.

Please test the following Docker image in your test environment and provide us some feedback:

edsiper/fluent-bit-1.3.9-next:1

cc: @rmacian

[rmacian]

looks good, my processing queue is being flushed

[edsiper]

Fixed on v1.3.9 release:

https://fluentbit.io/announcements/v1.3.9/