kafka plugin suggested updates

[kobihikri]

Hi all,

The kafka plugin is great and usable, but haven't got many updates to it recently.
In particular, I find it not to be a "fire and forget" connector, when considering ease of usage in a dynamic environment in which topics are dynamically added/removed.

To top that, it seems that the current connector version uses a legacy version of official kafka libraries (e.g., kafka-clients). Unfortunately, the used version has received many bug fixes and efficiency improvements that are not being used by Trino - and that's a shame ...

The last thing I would like to point out is that currently, the kafka plugin relies on an older version of Confluent's docker image for some tests. That means that different tests are running against different Confluent kafka docker images ...

Related to the last point - I'd propose to build a vanilla version of kafka and zookeeper images for testing purposes - versions that are matching the apache dependency versions the kafka plugin is using.

I would love to hear your comments on the matters I've added here, hoping we can begin a process of rewriting and modernizing the kafka plugin.

Related Issues:

#12739
#12740
#12826