Kubernetes integration

[otisg]

https://www.youtube.com/watch?v=lf31udm9cYY mentions Kubernetes integration and Kubernetes has a notion of Kubernetes Operator. It would be great if one could use CC with k8s.

[becketqin]

@otisg Yes, that is something valuable and it is on our road map. This may require some more thoughts and discussion though.

[otisg]

Right. Do you have a rough idea of the timeline?

[StevenACoffman]

@otisg You may wish to check out this kafka-operator which integrates cruise control.

Alternatively, this issue Yolean/kubernetes-kafka#100 seeks to integrate cruise control into the major Kubernetes Kafka implementation solution.

Unfortunately, there's currently a disconnect between the two approaches.

[becketqin]

@StevenACoffman Wow! This is awesome. Great to see the integration with K8S!

[baluchicken]

We've just released an open-source Kafka operator which integrates Cruise Control, and would be very happy to receive some feedback. You can check out the repo here.

[kyguy]

Since there are a few different options for running Kafka on Kubernetes, I think we should come up with a standard way of running Cruise Control on Kubernetes. For starters, we could provide a standard:

• Dockerfile
• Deployment
• ConfigMap for the cruisecontrol.properties file
• Service to expose the Cruise Control API

For what I understand, Cruise Control would also require a couple of patches for resource estimation and partition reassignment proposals while running on Kubernetes:

1. Currently, Cruise Control relies on the hostname of a Kafka broker e.g. node.host() to uniquely identify the node which that broker is running on [1]. However, assuming that every broker is run in its own Kubernetes pod, node.host() will return the hostname of the pod, not the hostname of the node that broker pod is running on. This leads Cruise Control to underestimate the resources available on nodes which are running more than one broker pod.
2. Since Cruise Control is not pod aware, it’s also possible that Cruise Control could propose partition assignments that place partition replicas on broker pods running on the same node. So if a node goes down, its possible we would lose more than one replica at once.

Using the Kubernetes API, we get around these issues by mapping broker pods to their actual nodes as Cruise Control builds its cluster model.

What do you all think? Any other ideas?

[1] https://github.com/linkedin/cruise-control/blob/2.0.70/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/LoadMonitor.java#L508
( Edited for static link )

[kyguy]

Would people be open to a patch that would:

1. Offer a boolean configuration option in the cruisecontrol.properties file that would specify whether or not Cruise Control would be running in a Kubernetes environment.
2. (and if the boolean option is set to true) Have a Kubernetes client use the hostname provided by node.host[1] to find the actual nodeName of where that broker host/pod is running from the Kubernetes API. Then use that nodeName as a unique identifier for the "hosts"(nodes) where the brokers reside when building the cluster model [2].

This would fix the two issues mentioned in the previous comment

[1] https://github.com/linkedin/cruise-control/blob/2.0.70/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/LoadMonitor.java#L508
[2] https://github.com/linkedin/cruise-control/blob/2.0.70/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/model/Rack.java#L259