Kafka scaler: concurrent offset fetches

[VerstraeteBert]

Concurrently query brokers for consumer and producer offsets

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated

Fixes #2377.

Signed-off-by: VerstraeteBert [email protected]

[zroubalik]

Looking good, I wonder if we can cover this with some test.

@pierDipi @lionelvillard @matzew mind looking?

[zroubalik]

@bpinske PTAL

[bpinske]

Looking good, I wonder if we can cover this with some test.

It looks like the kafka e2e tests already include multiple partitions. So I think this should be covered by existing tests. https://github.com/kedacore/keda/blob/main/tests/scalers/kafka.test.ts#L242

[VerstraeteBert]

Thanks for the feedback on my initial work by the way @bpinske

Found out that there is a goroutine leak when an error is thrown, fixed it. Tests don't cover the failure scenario, however, I do think it's sound now. A second opinion would be great though.

[bpinske]

The logic looks correct to me. As long as the e2e tests continue to work, I think this is a great improvement.

[PaulLiang1]

Looking good, I wonder if we can cover this with some test.

It looks like the kafka e2e tests already include multiple partitions. So I think this should be covered by existing tests. https://github.com/kedacore/keda/blob/main/tests/scalers/kafka.test.ts#L242

while i was working on #2409 , i had to setup the kafka e2e test locally, and it appears it runs a single node broker for the test.

for this change, would it be beneficial to cover multi broker scenario?

[pierDipi]

Looking good, I wonder if we can cover this with some test.

It looks like the kafka e2e tests already include multiple partitions. So I think this should be covered by existing tests. https://github.com/kedacore/keda/blob/main/tests/scalers/kafka.test.ts#L242

while i was working on #2409 , i had to setup the kafka e2e test locally, and it appears it runs a single node broker for the test.

for this change, would it be beneficial to cover multi broker scenario?

+1

[tomkerkhove]

Great to see a contribution from somebody in my neighborhood (also live near Bruges 😄 )

[VerstraeteBert]

Thanks for peeking @pierDipi, @PaulLiang1 . Have changed the the Strimzi cluster CR to spawn 3 brokers.

@tomkerkhove I see you're a howest alumni as well :-), cool! Could you trigger the e2e tests for me?

[JorTurFer]

/run-e2e kafka.test*
Update: You can check the progres here

[JorTurFer]

Hi @VerstraeteBert ,
Please sign you last commit, when you accept suggestions in GitHub, as default they are not signed :(
https://github.com/kedacore/keda/blob/main/CONTRIBUTING.md#i-didnt-sign-my-commit-now-what

[VerstraeteBert]

Might as well immediately concurrently fetch the consumer and producer offsets, while we're optimizing the metric generation. The I/O latency with the last commit changes from consumerT + ~max(producerTs) to ~max(consumerT, producerTs) .

What do you guys think about this change?

should I let getProducerOffsets and getConsumerOffsets take a resultChannel as a parameter instead since they are only used within a goroutine wrapper now?

[bpinske]

What do you guys think about this change?

I like it. The fewer round trips we need to wait on, the better.

should I let getProducerOffsets and getConsumerOffsets take a resultChannel as a parameter instead since they are only used within a goroutine wrapper now?

Personally I think it would be best to keep the function structured as you have. It may make future other feature improvements easier if concurrency isn't too tightly coupled to the logic of collecting offsets.

[VerstraeteBert]

What do you guys think about this change?

I like it. The fewer round trips we need to wait on, the better.

should I let getProducerOffsets and getConsumerOffsets take a resultChannel as a parameter instead since they are only used within a goroutine wrapper now?

Personally I think it would be best to keep the function structured as you have. It may make future other feature improvements easier if concurrency isn't too tightly coupled to the logic of collecting offsets.

Sounds good, thanks for the feedback @bpinske .

[zroubalik]

LGTM, thanks a lot @VerstraeteBert