KafkaChannel subscribers list ends up with duplicate subscribers for the same subscription. This results in duplicate dispatches.

[ntzlqx]

Describe the bug
Duplicate messages are dispatched to the same knative function

Expected behavior
Events are not duplicated

To Reproduce
I believe the issue occurs upon manual deletion of a subscription, but could be also upon upgrade (not verified)
Another way to replicate: Edit the broker config to add delivery options. This results in recreating all subscriptions and respectively all subscribers are recreated. You will see duplicate subscribers added to the subscribers list of the kafkachannel

Knative release version
0.17.3
Additional context
Editing manually the KafkaChannel subscribers list corrects the issue

[slinkydeveloper]

/assign

[slinkydeveloper]

@ntzlqx I tried to remove and create a sub again using a clean 0.17.3 setup, but i don't see any issue. Can you please provide a reproducer code/reproducer script?

Can you post the log (in debug mode) of kafka-ch-controller & kafka-ch-dispatcher when the issue happens?

[matzew]

I think knative/eventing#4042 is related here

/cc @pierDipi

[lionelvillard]

here more context: https://knative.slack.com/archives/CR2141CGY/p1599849495058400

[slinkydeveloper]

@matzew I'm not sure it's related to knative/eventing#4042, it seems more some corner case of the controller logic...

[slinkydeveloper]

Is this issue still relevant?

[ntzlqx]

I have not tested this...
The issue occurred when editing broker delivery section, I am going to do this today and see if duplicate subscribers appear

[slinkydeveloper]

Thanks!

[ntzlqx]

Just tested...and did get a duplicate set of subscribers...interestingly not a complete set...
I changed the backoffDelay and backoffPolicy on the broker
All subscriptions got recreated
A new set of subscribers appeared in the kafkaChannel subscribers list (with the new delivery config)
but "stale" subscribers(with the old delivery config) still linger on in the subscribers' list
I suspect this will result in duplicate event delivery as it did when I opened the issue

[slinkydeveloper]

Which version are you testing on?

[ntzlqx]

0.18

[ntzlqx]

I just manually deleted the stale ones - I have 20 subscribers - 7 remained as duplicates with the old config

[slinkydeveloper]

I'm not sure how i can help you without any reproducer, I'm sorry 😞 Can you try to dig into the logs of the kafka channel controller and of eventing-controller to check if there's anything suspect?

[ntzlqx]

Sorry for the delay...I will try to capture the log from the controller now

[ntzlqx]

The steps i am following:

1. Edit the the event delivery spec on the broker
2. Check the list of subscribers on the Kafka Channel for duplicates

[ntzlqx]

I got 8 duplicate subscribers out of 20 this time

[ntzlqx]

kafkaChController.log

Please find the attached log... I can reproduce the issue at will as described above

[slinkydeveloper]

@devguyio is this related to the status thing you've been working on recently? Can you check it please?

[devguyio]

This might get fixed by knative-extensions/eventing-kafka#182 once it lands.
/assign
/unassign @slinkydeveloper

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.