KAFKA-16368: Update default linger.ms to 5 for KIP-1030 #18080

jayteej · 2024-12-06T17:17:04Z

KAFKA-16368: As part of KIP-1030 we are increasing the default linger.ms to 5ms. System tests updated and behaviour validated manually.

divijvaidya · 2024-12-10T13:47:28Z

Similar comment as #18106 (comment). We need to be very explicit about the impact customers can expect on upgrade for this particular config.

jayteej · 2024-12-10T14:52:35Z

@divijvaidya done

divijvaidya

The change look good. I am waiting for the tests to succeed before merging this in.

divijvaidya

The change look good. I am waiting for the tests to succeed before merging this in.

jayteej · 2024-12-13T11:53:48Z

These tests are passing locally so I'm not sure whats going on. Will try another rebase to trunk

ijuma · 2024-12-22T06:49:30Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-                                                 + "batch size is under this <code>batch.size</code> setting.";
+                                                 + "This <code>linger.ms</code> setting defaults to 5, which means we'll wait 5ms for the accumulated "
+                                                 + "<code>batch.size</code> to fill as much as it can in this timeframe. This value previously defaulted"
+                                                 + "to 0, but over time we have observed that the IO overhead caused by smaller batches negates any latency gains.";


The wording here is a little odd. I would write something like:

"This defaults to 5, which means the producer will wait for 5ms or until the record batch is of batch.size (whichever happens first) before sending the record batch. Note that broker backpressure can result in a higher effective linger time than this setting.

The default changed from 0 to 5 in Apache Kafka 4.0 as the efficiency gains from larger batches typically result in similar or lower producer latency despite the increased linger."

Will update

ijuma · 2024-12-22T06:54:54Z

clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java

-                                                + "specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting <code>" + LINGER_MS_CONFIG + "=5</code>, "
-                                                + "for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.";
+                                                + "specified time waiting for more records to show up. This setting defaults to 5 (i.e. 5ms delay). Increasing <code>" + LINGER_MS_CONFIG + "=50</code>, "
+                                                + "for example, would have the effect of reducing the number of requests sent but would add up to 50ms of latency to records sent in the absence of load."


I don't follow, we seem to be saying that increasing from 5ms to 50ms would add up to 50ms of latency?

I have not changed the wording here. This is the same example with different values. I can change the wording also if you think it helps or isn't grokable?

ijuma · 2024-12-22T06:55:24Z

docs/upgrade.html

@@ -175,6 +175,9 @@ <h5><a id="upgrade_400_notable" href="#upgrade_400_notable">Notable changes in 4
                        </li>
                        <li>The deprecated <code>sendOffsetsToTransaction(Map&lt;TopicPartition, OffsetAndMetadata&gt;, String)</code> method has been removed from the Producer API.
                        </li>
+                        <li>The default value of <code>linger.ms</code> has been changed from 0ms to 5ms. The impact of this is larger request batches will be filled by default at the expense of
+                            time in the producer. It is generally observed that values lower than 5ms do not yield any latency benefits due to the IO overhead of smaller requests.


I would probably use similar wording to the one I suggested in a different comment.

Will update

divijvaidya · 2025-01-02T18:50:43Z

We pushed out a commit e099fce two weeks ago which should make the testBumpTransactionalEpochWithTV2Enabled stable. This is the test which is failing in this build.

@jayteej can you please rebase from trunk one more time.

jayteej · 2025-01-15T14:18:40Z

Known flakey failure

./gradlew clean :clients:test --tests 'org.apache.kafka.clients.ClientUtilsTest'

Gradle Test Run :clients:test > Gradle Test Executor 47 > ClientUtilsTest > testParseAndValidateAddressesWithReverseLookup() FAILED
    org.opentest4j.AssertionFailedError: Unexpected addresses [a96-7-128-175.deploy.static.akamaitechnologies.com, a23-215-0-138.deploy.static.akamaitechnologies.com, a23-215-0-136.deploy.static.akamaitechnologies.com, a23-192-228-84.deploy.static.akamaitechnologies.com, a23-192-228-80.deploy.static.akamaitechnologies.com, a96-7-128-198.deploy.static.akamaitechnologies.com, 2600:1408:ec00:36:0:0:1736:7f24, 2600:1406:bc00:53:0:0:b81e:94ce, 2600:1406:bc00:53:0:0:b81e:94c8, 2600:1406:3a00:21:0:0:173e:2e66, 2600:1406:3a00:21:0:0:173e:2e65, 2600:1408:ec00:36:0:0:1736:7f31] ==> expected: <true> but was: <false>
        at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
        at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
        at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
        at app//org.apache.kafka.clients.ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup(ClientUtilsTest.java:67)

Reviewers: Ismael Juma <[email protected]>, Divij Vaidya <[email protected]>

github-actions bot added core Kafka Broker producer clients small Small PRs labels Dec 6, 2024

jayteej force-pushed the KAFKA-16368-LINGER-MS branch 4 times, most recently from 05d0dd1 to 8308b32 Compare December 9, 2024 09:30

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from 8308b32 to 641e0f9 Compare December 10, 2024 14:51

divijvaidya approved these changes Dec 11, 2024

View reviewed changes

divijvaidya added the kip Requires or implements a KIP label Dec 11, 2024

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from 641e0f9 to a7b101e Compare December 12, 2024 11:07

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from a7b101e to 12f6bd8 Compare December 13, 2024 11:54

ijuma reviewed Dec 22, 2024

View reviewed changes

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from 12f6bd8 to 22f795c Compare December 23, 2024 15:33

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from 22f795c to dd07204 Compare January 3, 2025 10:49

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from dd07204 to d17b036 Compare January 15, 2025 10:09

divijvaidya added the ci-approved label Jan 15, 2025

KAFKA-16368: Update default linger.ms to 5 for KIP-1030

93019d4

jayteej force-pushed the KAFKA-16368-LINGER-MS branch from d17b036 to 93019d4 Compare January 15, 2025 15:13

divijvaidya approved these changes Jan 16, 2025

View reviewed changes

divijvaidya merged commit 11c10fe into apache:trunk Jan 16, 2025
9 checks passed

divijvaidya pushed a commit that referenced this pull request Jan 16, 2025

KAFKA-16368: Update default linger.ms to 5ms for KIP-1030 (#18080)

beda2f3

Reviewers: Ismael Juma <[email protected]>, Divij Vaidya <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-16368: Update default linger.ms to 5 for KIP-1030 #18080

KAFKA-16368: Update default linger.ms to 5 for KIP-1030 #18080

jayteej commented Dec 6, 2024

divijvaidya commented Dec 10, 2024

jayteej commented Dec 10, 2024

divijvaidya left a comment

divijvaidya left a comment

jayteej commented Dec 13, 2024

ijuma Dec 22, 2024 •

edited

Loading

jayteej Dec 23, 2024

jayteej Dec 23, 2024

ijuma Dec 22, 2024

jayteej Dec 23, 2024

ijuma Dec 22, 2024

jayteej Dec 23, 2024

jayteej Dec 23, 2024

divijvaidya commented Jan 2, 2025

jayteej commented Jan 15, 2025

KAFKA-16368: Update default linger.ms to 5 for KIP-1030 #18080

KAFKA-16368: Update default linger.ms to 5 for KIP-1030 #18080

Conversation

jayteej commented Dec 6, 2024

divijvaidya commented Dec 10, 2024

jayteej commented Dec 10, 2024

divijvaidya left a comment

Choose a reason for hiding this comment

divijvaidya left a comment

Choose a reason for hiding this comment

jayteej commented Dec 13, 2024

ijuma Dec 22, 2024 • edited Loading

Choose a reason for hiding this comment

jayteej Dec 23, 2024

Choose a reason for hiding this comment

jayteej Dec 23, 2024

Choose a reason for hiding this comment

ijuma Dec 22, 2024

Choose a reason for hiding this comment

jayteej Dec 23, 2024

Choose a reason for hiding this comment

ijuma Dec 22, 2024

Choose a reason for hiding this comment

jayteej Dec 23, 2024

Choose a reason for hiding this comment

jayteej Dec 23, 2024

Choose a reason for hiding this comment

divijvaidya commented Jan 2, 2025

jayteej commented Jan 15, 2025

ijuma Dec 22, 2024 •

edited

Loading