Skip to content

Commit

Permalink
in_kafka: Add buffer_max_size parameter instead of chunk_size
Browse files Browse the repository at this point in the history
We need to halt the greedy consuming loop before reaching the exceeded
limitation of chunk size.
Without this parameter, in_kafka always consuming greedy from
subscribing topics.

Signed-off-by: Hiroshi Hatake <[email protected]>
  • Loading branch information
cosmo0920 committed Nov 17, 2023
1 parent e0707cf commit a5c26ff
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pipeline/inputs/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
| client\_id | Client id passed to librdkafka. | |
| group\_id | Group id passed to librdkafka. | fluent-bit |
| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
| Chunk\_Size | Specify the size of buffer to store the incoming kafka messages. When this parameter is specified, do not poll them from brokers at once over the specified size and deferred to consume left messages from there at the next polling cycle. If not set, _Chunk\_Size_ is not effective. | |
| Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 5M |
| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
| rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | |

Expand Down

0 comments on commit a5c26ff

Please sign in to comment.