From c1e9527f02658faa53e34c5fc02de5b3be9daaef Mon Sep 17 00:00:00 2001 From: Denis Rechkunov Date: Wed, 26 Jan 2022 20:16:27 +0100 Subject: [PATCH] Document how timestamps set in Kafka messages affect retention (#30025) Some customers find the default behavior when using Kafka outputs unexpected.There are 2 possible options: 1. Message timestamp to be assigned on event creation (in beats), when the actual event was emitted and then propagated to Kafka 2. Message timestamp to be assigned when Kafka actually receives the message Since beats always set the message timestamp for Kafka version 0.10.0.0+ it's worth to document how to switch between the 2 options in Kafka configuration using the `log.message.timestamp.type` parameter. --- libbeat/outputs/kafka/docs/kafka.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libbeat/outputs/kafka/docs/kafka.asciidoc b/libbeat/outputs/kafka/docs/kafka.asciidoc index 28892f57aa01..a9b517d94903 100644 --- a/libbeat/outputs/kafka/docs/kafka.asciidoc +++ b/libbeat/outputs/kafka/docs/kafka.asciidoc @@ -11,6 +11,8 @@ To use this output, edit the {beatname_uc} configuration file to disable the {es output by commenting it out, and enable the Kafka output by uncommenting the Kafka section. +NOTE: For Kafka version 0.10.0.0+ the message creation timestamp is set by beats and equals to the initial timestamp of the event. This affects the retention policy in Kafka: for example, if a beat event was created 2 weeks ago, the retention policy is set to 7 days and the message from beats arrives to Kafka today, it's going to be immediately discarded since the timestamp value is before the last 7 days. It's possible to change this behavior by setting timestamps on message arrival instead, so the message is not discarded but kept for 7 more days. To do that, please set `log.message.timestamp.type` to `LogAppendTime` (default `CreateTime`) in the Kafka configuration. + Example configuration: [source,yaml] @@ -62,8 +64,6 @@ The cluster metadata contain the actual Kafka brokers events are published to. Kafka version {beatname_lc} is assumed to run against. Defaults to 1.0.0. -Event timestamps will be added, if version 0.10.0.0+ is enabled. - Valid values are all kafka releases in between `0.8.2.0` and `2.0.0`. See <> for information on supported versions.