[Kafka Scaler] Kafka SASL SCRAM auth seems to not work with AWS MSK

[dipen-epi]

Report

When trying to authenticate with AWS Managed Streaming for Kafka secured with SASL_SCRAM_512 authentication KEDA ScaledObject seems unable to communicate with cluster and gives the following error

{"ScaledObject.Namespace": "keda", "ScaledObject.Name": "kafka-scaledobject", "error": "error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

Tried using KEDA version 1.4 and 2.3.0.

Expected Behavior

KEDA should be able to create a new ScaledObject Custom Resource and subsequently a HPA for scaling the target Deployment

Actual Behavior

K8s cannot create the ScaledObject CR.

Steps to Reproduce the Problem

1. Kafka Secrets and TriggerAuthentication manifest

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: default
data:
  sasl: ""
  username: ""
  password: ""
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: default
spec:
  secretTargetRef:
  - parameter: sasl
    name: keda-kafka-secrets
    key: sasl
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password

2. ScaledObject manifest

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-msk-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-consumer
    # Required Name of the deployment we want to scale.
  pollingInterval: 10
  minReplicaCount: 0   #Optional Default 0
  maxReplicaCount: 10  #Optional Default 100
  advanced:                                          # Optional. Section to specify advanced options
    restoreToOriginalReplicaCount: true
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: b-1.xxx.kafka.us-east-1.amazonaws.com:9096,b-2.xxx.kafka.us-east-1.amazonaws.com:9096,b-3.xxx.kafka.us-east-1.amazonaws.com:9096
      consumerGroup: keda_scaling_topic_group
      topic: keda_scaling_topic
      lagThreshold: "30" 
    authenticationRef:
      name: keda-trigger-auth-kafka-credential

Logs from KEDA operator

{"ScaledObject.Namespace": "keda", "ScaledObject.Name": "kafka-scaledobject", "error": "error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

KEDA Version

2.3.0

Kubernetes Version

1.20

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

No response

[zroubalik]

What Kafka version are you using? You might have hit the same issue like mentioned here #1866?

[dipen-epi]

Looks like so @zroubalik
We're using Kafka version 2.7.0

[zroubalik]

2.7.0 should be supported by Sarama client that is being used by KEDA: https://github.com/Shopify/sarama#compatibility-and-api-stability

[dipen-epi]

Dynamic version support is part of KEDA release 2.4, if I'm not wrong?
We're currently using KEDA 2.3

[zroubalik]

In KEDA 2.4 you will be able to select the Kafka (Sarama) client version, in case you need to target an older Kafka Broker.

I think that your setup should work on KEDA 2.3 out of the box, because the Kafka version is supported by the client. I'd check whether the Kafka is reachable from within the cluster (firewall), I'd check if provided credentials and Broker urls are correct (check typos).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[djakielski]

This is stil an open issue when you try to use iam authentication. See IBM/sarama#1985

[syedzoherer]

I am facing same issue with AWS MSK when using SASL SCRAM. @dipen-epi is ur issue was resolved. Can u share what was the solution?
Kafka Version - 2.6.2
KEDA version - 2.6

[QualoZe0t]

i am using Keda app version 2.8.1 and kafka version 2.8.0 and getting same issue once AWS MSK is using SASL SCRAM

[joaopuccini]

i am using Keda app version 2.7.1 and kafka version 2.7.0 and getting same issue once AWS MSK is using SASL SCRAM

[JorTurFer]

Maybe it's a super obvious question but, are you using scram_sha512? I mean, in theory based on docs, SCRAM should work properly. Are you using tls?

[QualoZe0t]

I have also active issue where sasl ( scram_sha512 ) is not working in our case with tls but without ACM configuration.
issue: #4089

[JorTurFer]

TBH, I have no idea about AWS MSK, yesterday I created a cluster but I couldn't expose it publicly. For troubleshooting proposes I need to reach the kafka cluster from my local, which was impossible. I'll give another try when I have some time

[JorTurFer]

To not discuss in a closed issue, let's continue the conversation in the new issue you have created #4089