Kafka consumers scaling from zero with timed messages

[sansmoraxz]

Say I have a scaler defined something like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-scaler-1
spec:
  scaleTargetRef:
    name: my-statefulset-1
    kind: StatefulSet
  pollingInterval: 5
  cooldownPeriod: 300
  idleReplicaCount: 0
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: "kafka-broker-1.kafka-endpoints.svc.cluster.local:9092,kafka-broker-2.kafka-endpoints.svc.cluster.local:9092"
      consumerGroup: MyConsumerGroup
      topic: consumer.data
      lagThreshold: "10000"
      offsetResetPolicy: latest
      activationLagThreshold: "1"

The pods in my-statefulset-1 have auto.offset.reset as latest

Let's say the topic consumer.data is newly created and no offsets has been commited yet. The pods are scaled to minReplicaCount which is to be expected. And after 5 mins of empty queue, the pods will be scaled down to 0.

Now let's assume that the topic consumer.data is set to clear old data with log.retention.hours=1

If a new record is added to the queue, after 1 hour; the scaler occasionally refuses to scale out, with no errors being logged, and status as "Scaling is not performed because triggers are not active"

This behavior sticks even after the scaler is recreated. Hell even manually scaling the pod starts to scale it back down to zero.

So any suggestions on what I may be missing?

keda version: 2.10.0
kafka version: 3.3.2

[zroubalik]

What if you remove activationLagThreshold: "1" from your SO?

[sansmoraxz]

Will that not disable 0-1 scaling and hence idle value?

[zroubalik]

Just use minReplicaCount:0 & maxReplicaCount: 10 and remove idleReplicaCount. It will properly scale between 0-10. idleReplicaCount is only useful when you want to scale from 0 directly to some min value greater than 1.

[sansmoraxz]

OK weirdly enough this works.

[sansmoraxz]

Still not sure why this was happening though. Prometheus and other metrics captures were still reporting offset lags as 1 for the group.

[ganesh-kr]

Hey can someone help me with this

We are encountering an issue with our KEDA scaled object triggers configuration. When there is a lag on the configured Kafka topic, the metrics value is being displayed as "673m" and "3122m" as the lag increases. Ideally, it should represent the number of messages, but it's being shown in this format.

This issue is happening with the Kafka scaler when used in conjunction with CPU and memory metrics. However, when used separately, the Kafka scaler works as expected, showing the value as the number of messages.

We need assistance in understanding why the values are represented in this manner and whether it's possible to use a combination of CPU, memory, and Kafka scalers, or if any modifications are needed in the configuration.

Below is my scaled object configuration

triggers:

• metadata:
  bootstrapServers: ip:port
  consumerGroup: group_name
  lagThreshold: '10'
  offsetRestPolicy: latest
  topic: topic_name_1
  type: kafka
• metadata:
  bootstrapServers: ip:port
  consumerGroup: group_name
  lagThreshold: '10'
  offsetRestPolicy: latest
  topic: topic_name_2
  type: kafka
• metadata:
  containerName: xyz
  value: '60'
  metricType: Utilization
  type: cpu
• metadata:
  containerName: xyz
  value: '60'
  metricType: Utilization
  type: memory

HPA metrics output
Metrics: ( current / target )
resource cpu of container "xyz" on pods (as a percentage of request): 4% (573m) / 60%
resource memory of container "xyz" on pods (as a percentage of request): 58% (16434180096) / 60%
"s0-kafka-topic_name_1" (target average value): 1522m / 10 ( exact lag is around 32 messages )
"s1-kafka-topic_name_2" (target average value): 672m / 10