Configuring kafka producer clients

[lftrevisan]

Hi folks,

We've been trying to configure a kafka cluster using kafka mesos framework and we're having some problems in order to decide how to configure our kafka java producer client.

It requires the bootstrap.servers argument that should point to a list of broker IPs for the initial metadata information but now the brokers may be running on any instance of my mesos cluster.

I understand that I can query the kafka scheduler REST API and get a list of brokers and then dynamically configure my producer client or we can also run Mesos DNS and use DNS as well.

I'd like to get some ideas how you have been configuring your producer client.

Luiz

[gondor]

+1 Would love to see the teams thoughts on this too, going through the same process.

[joestein]

We have been talking about adding to the scheduler interface the bootstrap response for beta data. This way the scheduler should "talk" like a broker and be able to then have the producer get the information the scheduler wants to send for which broker partition to talk too. Its the type of things we have done in custom implementations you can do blue green with monkey patching the min broker call https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L352-L370 in the producer for which broker to use (current producer kafka does min requests, this makes it talk back to broker proxy in this OSS case would be scheduler).

We have been working on this https://github.com/elodina/java-mesos-util so we can support both "old" and "new" Mesos API (1.0 is coming) so maybe after that and a few other things we can get back this items (it is really near on our roadmap)

[ghughal]

+1 for this. We are having similar issue as well.

[steveniemitz]

+1 also, this is a great idea!