Outside (i.e. cluster-external) access through per-broker Service

[solsson]

This is my interpretation of a baseline for different type of clusters, from #13.

The new pod label is low-risk, as are the services. The new directives in server.properties need more testing. Might de-stabilize regular inside access.

I've tested this only on minikube so far:

• Internal access still works, without changes, according to test/basic-with-kafkacat.yml
• Outside access:

BOOTSTRAP=$(minikube ip):32400,$(minikube ip):32401,$(minikube ip):32402
docker run --rm -t solsson/kafkacat -C -b $BOOTSTRAP -t test-basic-with-kafkacat -o -10

Outside+minikube is interesting as a local development setup for Kafka services. Production clusters on the other hand will probably always need tailored advertised.listeners (maybe through the host lookup command in init config) and listener.security.protocol.map.

[solsson]

@comdw in response to #13 (comment), feel free to test with this PR. It works on minikube, and quite possibly also on GKE with outside meaning TCP access to the nodes' internal IPs. In particular I'm interested in how to adapt the init commands to different environments.

[comdw]

@solsson - tested this and it works fine for me.

[yacut]

@solsson works for me. Thanks!

[codyAnivive]

@solsson works great on bare metal deployment as well

[solsson]

I'm also using this now. Thanks for the great discussion in #13.

[StevenACoffman]

I wrote this to test the outside services:

function join_by { local IFS="$1"; shift; echo "$*"; }

BOOTSTRAP_ARRAY=()
POD_NAME_ARRAY=("kafka-0" "kafka-1" "kafka-2")
for POD_NAME in "${POD_NAME_ARRAY[@]}"
do
  BOOTSTRAP_IP=$(kubectl get pods ${POD_NAME} -n kafka -o jsonpath='{.metadata.labels.kafka-listener-outside-host}')
  BOOTSTRAP_PORT=$(kubectl get pods ${POD_NAME} -n kafka -o jsonpath='{.metadata.labels.kafka-listener-outside-port}')
  BOOTSTRAP_ARRAY+=("$BOOTSTRAP_IP:$BOOTSTRAP_PORT")
done

BOOTSTRAP=$(join_by , "${BOOTSTRAP_ARRAY[@]}")
docker run --rm -t solsson/kafkacat -C -b $BOOTSTRAP -t test-basic-with-kafkacat -o -10

[solsson]

^ Nice example of how to use the new labels.

[solsson]

Based on the above I found a horrible :) one-liner relying on bootstrapping, to test #120: docker run --rm solsson/kafkacat -C -b $(kubectl -n kafka get pod kafka-0 -o go-template --templat'{{index .metadata.labels "kafka-listener-outside-host"}}:{{index .metadata.labels "kafka-listener-outside-port"}}') -t heapster-metrics

[StevenACoffman]

That is either terrible, or wonderful. I guess both. 😄

[mtbbiker]

Just an idea that I have managed to use to get the "outside" service to "connect" to the "Pod'

kind: Service
apiVersion: v1
metadata:
  name: outside-0
  namespace: kafka
spec:
  selector:
    app: kafka
    statefulset.kubernetes.io/pod-name: kafka-0
  ports:
  - protocol: TCP
    targetPort: 9093
    port: 9093

On previous versions of K8S I also used @solsson initContainer idea. However in version 1.11.1 I couldn't get it to work to create the labels . So I got it to work by adding statefulset.kubernetes.io/pod-name: kafka-0 to the selector: spec section
Add this now for each outside service by incrementing the value for kafka-1, kafka-2 etc.

[solsson]

@mtbbiker Can you please create an issue for the errors with k8s 1.11.1, with the output from kubectl -n kafka logs kafka-0 -c init-config?

[mtbbiker]

@solsson As requested for the zookeeper deployment, the logs

kubectl -n kafka logs pzoo-0 -c init-config
1
+ '[' -z '' ']'
+ ID_OFFSET=1
+ export ZOOKEEPER_SERVER_ID=1
+ ZOOKEEPER_SERVER_ID=1
+ echo 1
+ tee /var/lib/zookeeper/data/myid
+ sed -i 's/server\.1\=[a-z0-9.-]*/server.1=0.0.0.0/' /etc/kafka/zookeeper.properties
sed: can't read /etc/kafka/zookeeper.properties: No such file or directory

I am busy with a new deployment and log a seperate Issue

[mtbbiker]

@solsson I found a issue in the ConfigMap yaml file, I have Zookeeper working now