Skip to content

Commit

Permalink
Drop experimental label for rebalanceSafeCommits (#1222)
Browse files Browse the repository at this point in the history
RebalanceSafeCommits mode has been used extensively in production and
did not lead to any problems. I think it's time to remove the
experimental label.
  • Loading branch information
erikvanoosten authored Apr 18, 2024
1 parent 2149034 commit 09c6266
Show file tree
Hide file tree
Showing 3 changed files with 32 additions and 4 deletions.
31 changes: 31 additions & 0 deletions docs/preventing-duplicates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
id: preventing-duplicates
title: "Preventing duplicates"
---

In zio-kafka processing of records runs asynchronously with partition management. This brings substantial performance
advantages but causes some records to be consumed and processed _twice_ when a rebalance occurs. To prevent this,
since version 2.7.1 zio-kafka supports a new mode in which we prevent duplicates due to rebalances. You can enable it
as follows:

```scala
import zio.*
import zio.kafka.consumer.ConsumerSettings

val consumerSettings: ConsumerSettings =
ConsumerSettings(List("localhost:9092"))
.withGroupId("group")
.withRebalanceSafeCommits(true) // enable rebalance-safe-commits mode
.withMaxRebalanceDuration(30.seconds) // defaults to 3 minutes
```

With rebalance-safe-commits mode enabled, rebalances are held up for up to max-rebalance-duration to wait for pending
commits to be completed. Once pending commits are completed, it is safe for another consumer in the group to take over
a partition.

For this to work correctly, your program must process a chunk of records within max-rebalance-duration. The clock
starts the moment the chunk is pushed into the stream and ends when the commits for these records complete.

For more information see the scaladocs in `ConsumerSettings`, read the description of
[pull request #1098](https://github.com/zio/zio-kafka/pull/1098) that introduced this feature, or watch the presentation
[Making ZIO-Kafka Safer And Faster](https://www.youtube.com/watch?v=MJoRwEyyVxM). The relevant part starts at 10:24.
1 change: 1 addition & 0 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ const sidebars = {
"example-of-consuming-producing-and-committing-offsets",
"partition-assignment-and-offset-retrieval",
"consumer-tuning",
"preventing-duplicates",
"sharing-consumer",
"serialization-and-deserialization",
"writing-tests"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,6 @@ final case class ConsumerSettings(
copy(restartStreamOnRebalancing = value)

/**
* NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.
*
* @param value
* Whether to hold up a rebalance until all offsets of consumed messages have been committed. The default is
* `false`, but the recommended value is `true` as it prevents duplicate messages.
Expand Down Expand Up @@ -224,8 +222,6 @@ final case class ConsumerSettings(
copy(rebalanceSafeCommits = value)

/**
* NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.
*
* @param value
* Maximum time spent in the rebalance callback when `rebalanceSafeCommits` is enabled.
*
Expand Down

0 comments on commit 09c6266

Please sign in to comment.