Skip to content

Commit

Permalink
Document how timestamps set in Kafka messages affect retention (elast…
Browse files Browse the repository at this point in the history
…ic#30025)

Some customers find the default behavior when using Kafka outputs
unexpected.There are 2 possible options:

1. Message timestamp to be assigned on event creation (in beats), when the actual event was emitted and then propagated to Kafka
2. Message timestamp to be assigned when Kafka actually receives the
message

Since beats always set the message timestamp for Kafka version 0.10.0.0+ it's
worth to document how to switch between the 2 options in Kafka
configuration using the `log.message.timestamp.type` parameter.
  • Loading branch information
rdner authored and yashtewari committed Jan 30, 2022
1 parent 170e1ab commit c1e9527
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions libbeat/outputs/kafka/docs/kafka.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ To use this output, edit the {beatname_uc} configuration file to disable the {es
output by commenting it out, and enable the Kafka output by uncommenting the
Kafka section.

NOTE: For Kafka version 0.10.0.0+ the message creation timestamp is set by beats and equals to the initial timestamp of the event. This affects the retention policy in Kafka: for example, if a beat event was created 2 weeks ago, the retention policy is set to 7 days and the message from beats arrives to Kafka today, it's going to be immediately discarded since the timestamp value is before the last 7 days. It's possible to change this behavior by setting timestamps on message arrival instead, so the message is not discarded but kept for 7 more days. To do that, please set `log.message.timestamp.type` to `LogAppendTime` (default `CreateTime`) in the Kafka configuration.

Example configuration:

[source,yaml]
Expand Down Expand Up @@ -62,8 +64,6 @@ The cluster metadata contain the actual Kafka brokers events are published to.

Kafka version {beatname_lc} is assumed to run against. Defaults to 1.0.0.

Event timestamps will be added, if version 0.10.0.0+ is enabled.

Valid values are all kafka releases in between `0.8.2.0` and `2.0.0`.

See <<kafka-compatibility>> for information on supported versions.
Expand Down

0 comments on commit c1e9527

Please sign in to comment.