Dapr client stopped processing events from event hub after a long idle time

[skbar50]

Problem description

We are facing an issue with dapr. One of our application subscribes to an event hub. Recently we are observing the application stops receiving events from the event hub. The issue doesn't occur regularly. This usually happens if there is a long gap after the last event received.

The pod has some other functionalities which are working fine. Other pods make api calls running on this pod through dapr. That also works fine. Only the receiving of events doesn't work until we restart.

We are using dapr version 1.10.4.

What version of Dapr?

1.10.4

[ItalyPaleAle]

@skbar50 could you please try with Dapr 1.11.1?

In the 1.11 branch we are using a newer, stable SDK for Azure Event Hubs and I suspect that will have fixed a lot of issues already.

[skbar50]

Thank you for your replies. We are in the process of upgrading to Dapr 1.11.1. I will let you know how this works out for us.

[berndverst]

@skbar50 we are aware that Dapr 1.10.X may suffer from authentication tokens expiring after ~7 days which can cause events to not be processed. A decision was made not to backport this fix to 1.10.X in Dapr open source - though the fix has been available in Azure Container Apps and and the AKS extension with Dapr 1.10.8. Any Dapr 1.11.X release should mitigate this issue.

We included another EventHubs improvement in 1.11.2 that was just released (though we have never seen the issue this improves in the wild). I strongly recommend upgrading to 1.11.2 from 1.10.X due to massive amounts of improvements in stability for EventHubs (caused by switching to a completely Storage SDK, and several significant version upgrades of the EventHubs SDKs and amqp SDKs)

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[ItalyPaleAle]

Fixed by #3344