Awaiting device stream request inside of direct method handler

[Perun85]

• OS, version, SKU and CPU architecture used: Windows 10 Pro
• Application's .NET Target Framework : net471
• Device: Laptop
• SDK version used:
  Microsoft.Azure.Devices.Client 1.29.0-preview-002
  Microsoft.Azure.Devices 1.27.0-preview-003
• Protocol: MQTT

Description of the issue:

When trying to await for WaitForDeviceStreamRequestAsync() inside of direct method handler deviceClient gets disconnected.
We use the flow that involves direct methods, because we need to pass additional information (IP address, port, etc.).
Is it even possible to use it in this way?

Code sample exhibiting the issue:

        deviceClient.SetConnectionStatusChangesHandler((state, reason) => {
             Console.WriteLine($"Connection state: {state}, reason: {reason}.");
       });

        MethodCallback methodCallback = async (request, state) => {
           
        await deviceClient.WaitForDeviceStreamRequestAsync().ConfigureAwait(false); // when this code section is reached, device disconnects
            return new MethodResponse(200);
        };

        await deviceClient.SetMethodHandlerAsync("TestMethod", methodCallback, null).ConfigureAwait(false);

        await deviceClient.OpenAsync().ConfigureAwait(false);

Console log of the issue:

Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttIotHubAdapter.d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttIotHubAdapter.d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.d__281.MoveNext() --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<ExecuteWithErrorHandlingAsync>d__281.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass23_0.<b__0>d.MoveNext()

[Perun85]

Today, I have tested it on IoTHub from S1 tier (previous error is from IoTHub from Free tier ). When using S1 instance I was able to wait for device stream request inside direct method handler. After I accepted the request and shut down the application I was not able to reconnect again to IoTHub instance from that device any more. I was constantly getting

Connection state: Disconnected, reason: Retry_Expired

Only solution was to delete device and recreate it.

Afterwards, I set custom retry policy in order to catch following exception and stack trace.

Exception message:
Transient network error occurred, please retry.

StackTrace:
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<ExecuteWithErrorHandlingAsync>d__28 1.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass41_0.<<OpenAsyncInternal>b__0>d.MoveNext()

In both hubs (S1 and F1) devices where authenticated with self-signed certificates.

[Perun85]

It looks like when MQTT protocol is used, if we even just have registered direct methods (we are not combining them with device streams workflow), and we try to wait for device stream request, after client connection is once closed, it cannot be established again with that same device.

[vinagesh]

@Perun85 - Do you still need help on this? Can you give more details on the scenario?

[Perun85]

I haven't tried this in a while. We switched to solution where we use two IoT Hubs, one only for device streams and other for everything else. When device streams become GA, we plan to move everything again to just one IoT Hub and it would be great if this bug can be resolved by then.

[drwill-ms]

Moving all device streaming issues to this discussion #1811.