Update Kafka dependencies to prepare for KIP-896

[nhaq-confluent]

Hello!

I am from Confluent, an organization contributing to Apache Kafka, and wanted to raise a request in regards to KIP-896.

In short, KIP-896 removes compatibility of certain client protocol API versions in Apache Kafka(AK) 4.0, being targeted for around January 2025. Since there is a dependency on Kafka in beats (in particular to a forked version of the Sarama client) I wanted to raise awareness that there could be some changes necessary to ensure a smooth experience for those using beats and Kafka. Here are the recommendations.

1. The latest versions of beats (7.X and higher) appear to be using sarama 1.29.1, which is good enough from the version perspective, however the Kafka version would need to be set to 2.1.0 manually by users to make this work. Recommend updating the default to be 2.1.0 or higher.

2. Recommend updating the sarama dependency to the latest version possible in future beats versions

3. Updating the beats docs, advising users of this situation and to set the Kafka version config to 2.1.0

Thank you!

[ijuma]

Reference to the PR where the default for Sarama was changed: IBM/sarama#2574

[nhaq-confluent]

@ruflin @andrewkroh could you please comment on this issue?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

Hello @nhaq-confluent
Thanks for raising this issue and bringing some context to it.
We are currently discussing about what would be the best timing to change the default version to 2.1.0 as suggested.

[ijuma]

Thanks. One important detail is that 2.1.0 is truly ancient at this point, it was released 6 years ago (in Nov 2018). So, we should consider changing the default for older releases too to avoid the situation where they are completely broken with Apache Kafka 4.0 (due in 2-3 months).

[khushijain21]

Hello @nhaq-confluent @ijuma, is it possible to have access to 4.0.0 SNAPSHOT version for testing? This would help us perform upgrades and test it locally in our labs. TIA

[khushijain21]

Closing this as completed by

upgrades sarama version on elastic fork - elastic/sarama#19
Update beats to latest sarama version - #41655
set default kafka protocol version 2.1.0 - #41662
upgrades the documentation - #41540