Apache Kafka Scaler: Implementation for Excluding Persistent Lag

[josephangbc]

Proposal

Add implementation for excluding consumer lag from partitions with persistent lag.

Use-Case

In situations where consumer is unable to process / consume from partition due to errors etc., committed offset will not change, and consumer lag on that partition will be increasing and never be decreased. KEDA trigger scaling towards the maxReplicaCount.

If partition lag is deemed as persistent, excluding its consumer lag will allow KEDA to trigger scaling appropriately based on the consumer lag observed on other topics and partition, and not be affected by this consumer lag which will not be resolved by scaling.

Anything else?

Implementation I have in mind is as follows:
Upon each polling cycle, check if current consumer offset is same as previous consumer offset.

• Different: return endOffset - consumerOffset (No different from current implementation)
• Same: return 0 (To exclude this partition's consumer lag from the total lag)

I have a working proof-of-concept for this implementation, will add a PR to this feature request issue if neccessary. Please let me know your thoughts / comments about the usefulness of this proposal and also about the implementation I suggested above.

Pull Requests

keda: #3965
keda-docs: kedacore/keda-docs#984