From c6f5a184969b6d31740855d0a77c77bf90de5bff Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 17 Nov 2023 20:19:46 +0900 Subject: [PATCH] in_kafka: Add buffer_max_size parameter instead of chunk_size We need to halt the greedy consuming loop before reaching the exceeded limitation of chunk size. Without this parameter, in_kafka always consuming greedy from subscribing topics. Signed-off-by: Hiroshi Hatake --- pipeline/inputs/kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index 78610b11d..caf93f2cc 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -12,7 +12,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | -| Chunk\_Size | Specify the size of buffer to store the incoming kafka messages. When this parameter is specified, do not poll them from brokers at once over the specified size and deferred to consume left messages from there at the next polling cycle. If not set, _Chunk\_Size_ is not effective. | | +| Buffer\_Max\_Size | Specify the maximum size of buffer to store the incoming kafka messages. To increase throughput, specify larger size. | 5M | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | | rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | |