[Question] Strimzi Kafka Bridge inside Service Mesh

[srinivasev]

Hi Team,

I am trying to enable the Strimzi Bridge inside the Service Mesh. For this, I am making the below changes in KafkaBridge yaml file. You can see the annotations added related to service mesh.

Actually the plan is to have a side car container along with Strimzi Bridge pod which takes care of encryption of traffic moving to Strimzi Kafka which is outside the service mesh. Along with encryption, plan is to gain other Istio related benefits.

`apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaBridge
metadata:
name: kf-bridge
labels:
{{- include "kf-bridge-sz.labels" . | nindent 4 }}
annotations:
{{- include "kf-bridge-sz.annotations" .| nindent 4 }}
prometheus.io/path: /metrics
prometheus.io/port: "{{ .Values.bridgeport }}"
prometheus.io/scrape: "true"
spec:
template:
pod:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: 8080
prometheus.io/scrape: "true"
sidecar.istio.io/inject: true
sidecar.istio.io/userVolume: {"test-oss-dmm-kf-certs-tls":{"secret":{"secretName":"test-oss-dmm-kf-secret","optional":true}},"test-oss-dmm-kf-certs-ca-tls":{"secret":{"secretName":"test-sec-sip-tls-trusted-root-cert"}}}
sidecar.istio.io/userVolumeMount: {"test-oss-dmm-kf-certs-tls":{"mountPath":"/etc/istio/tls/test-oss-dmm-kafka-bootstrap/","readOnly":true},"test-oss-dmm-kf-certs-ca-tls":{"mountPath":"/etc/istio/tls-ca","readOnly":true}}
labels:
sidecar.istio.io/inject: true
bootstrapServers: kf-bootsrap-server:9093
http:
port: 8080
image: {{ template "kf-bridge-sz.imagePath" (dict "imageId" "kf-bridge-sz" "files" .Files "values" .Values) }}

`

The below block is not added intentionally so that bridge will not take care any tls encryption or truststore related load on it and entirely depends on side car for this.

authentication:
type: tls
certificateAndKey:
secretName: test-oss-kf-bridge-secret
certificate: cert.pem
key: key.pem
tls:
trustedCertificates:
- secretName: test-tls-trusted-root-cert-ca
certificate: ca.crt

Currently with this change, we see the requests are timing out in Strimzi Bridge and unable to hit the Strimzi Kafka.

Strimzi bridge pod logs
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata 2022-10-19 06:47:28 INFO listTopics:85 - [448713620] LIST_TOPICS Request: from 127.0.0.6:44569, method = GET, path = /topics 2022-10-19 06:47:28 INFO AdminClientEndpoint:88 - List topics 2022-10-19 06:47:28 INFO listTopics:85 - [448713620] LIST_TOPICS Response: statusCode = 200, message = OK 2022-10-19 06:47:51 INFO AdminMetadataManager:235 - [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1666162070917, tries=1, nextAllowedTryMs=1666162071149) timed out at 1666162071049 after 1 attempt(s) Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata 2022-10-19 06:48:21 INFO AdminMetadataManager:235 - [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1666162101049, tries=1, nextAllowedTryMs=1666162101202) timed out at 1666162101102 after 1 attempt(s)

Strimzi Kafka pod logs - ( 192.168.244.24 is bridge pod IP, 192.168.123.67 is kafka pod IP).
2022-10-19 06:51:15,313 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /192.168.244.24 (channelId=192.168.123.67:9093-192.168.244.24:52606-24636) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(TLS-9093)-SSL-9] 2022-10-19 06:51:16,732 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /192.168.244.24 (channelId=192.168.123.67:9093-192.168.244.24:52726-24637) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(TLS-9093)-SSL-7] 2022-10-19 06:51:18,137 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /192.168.244.24 (channelId=192.168.123.67:9093-192.168.244.24:52896-24637) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(TLS-9093)-SSL-8] 2022-10-19 06:51:19,554 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /192.168.244.24 (channelId=192.168.123.67:9093-192.168.244.24:53052-24637) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(TLS-9093)-SSL-9] 2022-10-19 06:51:20,952 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /192.168.244.24 (channelId=192.168.123.67:9093-192.168.244.24:53218-24638) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(TLS-9093)-SSL-7]

Please let us know on this issue.

[srinivasev]

Hi Team,
Please let me know if there is any update on this.

[scholzj]

Ad I said on Slack, Strimzi does not have any support for Istio. You might get something working, but you have to figure it your self.

[srinivasev]

Hi @scholzj ,
Thank you.
From the error, can you guide me anything is possible.

Just posted this question if any consumers of Strimzi tried istio which might help me.