From 0e548b72170b0a89790cdd2390040588dfbf0ada Mon Sep 17 00:00:00 2001 From: Thiago Padilha Date: Thu, 4 Jan 2024 19:54:30 +0000 Subject: [PATCH] input: kafka: Document `data_format` option (#1118) * input: kafka: Document `data_format` option Signed-off-by: Thiago Padilha * Rename data_format to format Signed-off-by: Thiago Padilha --------- Signed-off-by: Thiago Padilha --- pipeline/inputs/kafka.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index 358a46bc5..e779b6f07 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -9,6 +9,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill | :--- | :--- | :--- | | brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | | | topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | | +| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none | | client\_id | Client id passed to librdkafka. | | | group\_id | Group id passed to librdkafka. | fluent-bit | | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | @@ -53,6 +54,7 @@ The fluent-bit source repository contains a full example of using fluent-bit to brokers kafka-broker:9092 topics fb-source poll_ms 100 + format json [FILTER] Name lua @@ -68,6 +70,8 @@ The fluent-bit source repository contains a full example of using fluent-bit to The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds. +Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`. + Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker. The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used).