Stream Removed

[Lamby123]

Describe the bug
Several times a day we get a "stream removed" error coming back from the Zeebe Client.

One or more errors occurred. (Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: ................

To Reproduce
Steps to reproduce the behavior:

1. Make thousands of requests per day with the Zeebe client. Eventually one will fail with this error message and will subsequently be successful on repeated attempts.

Expected behavior
It looks like this is a known issue with certain versions of GRPC core libaries.
We would expect this error to be handled within the client and for the request to be retried, or for the version of GRPC to be updated to one that does not have this issue . (if there is one).

Enviroment (please complete the following information):

• Reproduced with Versions of Zeebe: 0.23.1, 0.24.3, 0.25.1

[jwulf]

This happens in the Node client as well. Does the C# client use the Google C client under the hood?

[ChrisKujawa]

Actually this issue was covered also by a support case, for reference https://jira.camunda.com/browse/SUPPORT-8433

It seems to be related to grpc/grpc#15623 and grpc/grpc-node#708

[ChrisKujawa]

I don't think that we can do anything here. If you want to retry these kind of calls your could try the new sendWithRetry methods, see https://github.com/camunda-community-hub/zeebe-client-csharp/blob/master/Client/Api/Commands/IFinalCommandWithRetryStep.cs Be aware that not all commands implement that yet

[ChrisKujawa]

Regarding your question @jwulf it seems so, output from the user agent: zeebe-client-csharp/0.19.0.0 grpc-csharp/2.37.1 grpc-c/15.0.0 (linux; chttp2)

[ChrisKujawa]

I think we could overcome this with switching to dotnet-grpc, which is available for .net core 3.0 https://grpc.io/blog/grpc-on-dotnetcore/

The question would be if we want to drop core 2.1

[ChrisKujawa]

@Lamby123 which .net version do you use? Would be migrating the client to .net core 3.x or higher suitable to you?