[BUG] Eventhub SDK 5.10.0 stops reading Partitions

[the-mod]

Describe the bug
I am using EventProcessorClient from azure-messaging-eventhubs:5.10.0 together with azure-messaging-eventhubs-checkpointstore-blob:1.9.0 to read Events from Eventhubs.

The reading of Partitions stops out of sudden and does not recover.
This leads to the fact, Events on the affected Partitions are piling up.
Only a restart of the Application mitigates this Issues, but after some time it hit again and again.

Exception or Stack Trace
In the logs I see ReactorDispatcher instance is closed Exceptions coming from the SDK.
Unfortunately no Stacktrace is provided

To Reproduce
Found no way to reproduce due to the fact it hit my Services at random uptime.
It seams also to be independent on the amount of Events read out by the App.

Code Snippet
This is my current way to init a EventProcessorClient:

new EventProcessorClientBuilder()

.connectionString(connString)

.consumerGroup(cg)

.checkpointStore(new BlobCheckpointStore(containerClient))

.loadBalancingUpdateInterval(Duration.ofSeconds(5))

.partitionOwnershipExpirationInterval(Duration.ofSeconds(20))

.loadBalancingStrategy(LoadBalancingStrategy.GREEDY)

.processEventBatch(consumer, batchSize)

.processError(errorConsumer)

.processPartitionInitialization(initConsumer)

.processPartitionClose(closeConsumer)

.buildEventProcessorClient();

Expected behavior
Run for ever and ever

Setup (please complete the following information):

• OS: Windows
• Library/Libraries: azure-messaging-eventhubs:5.10.0 and azure-messaging-eventhubs-checkpointstore-blob:1.9.0
• Java version: azul JRE 16
• App Server/Environment: Tomcat 9.0.52
• Frameworks: Spring Boot 2.4.10

[joshfree]

@conniey could you follow up with @the-mod on this Event Hubs ReactorDispatcher instance is closed issue?

/cc @anuchandy

[conniey]

Hey, the ReactorDispatcher instance is closed is misleading (and we plan on suppressing this because it isn't actually the cause of errors). I was wondering if you had any logs so we can see what happened before this exception.

[edit] We did fix a bug this past two weeks where a graceful close from the service would result in the receives not resuming... I'd need to look at the logs to understand if its the issue you're running into though. #24141

[the-mod]

@conniey unfortunately nothing more than that in the timerframe some readers stopped working

[paalkristian]

We are experiencing this behavior as well. We are using azure-messaging-eventhubs:5.10.0 with azure-spring-cloud-stream-binder-eventhubs:2.8.0, which is using azure-messaging-eventhubs-checkpointstore-blob:1.9.0. Our application runs without problems between 4 and 14 days, before some messages are no longer consumed.

When I restart the consumer it works as expected again. I might be able to provide logs if it helps.

[conniey]

@paalkristian Hey. Providing logs would help because the end result of most bugs is that it stops receiving, but the cause is often different.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!