Dotnet-monitor hangs for apps spawned in the background

[wiktork]

When creating apps in the background and using the hosting process (e.g. dotnet run &), dotnet-monitor will hang when making rest calls. So far, this has been reproduced on WSL2 with Ubuntu 18.04 and 20.04.

[jander-msft]

Example callstack of hang:

System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle socket, System.Net.Sockets.SocketFlags flags, System.Span<byte> buffer, byte[] socketAddress, ref int socketAddressLen = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out Interop.Error errno = SUCCESS)
System.Net.Sockets.dll!System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle socket, System.Span<byte> buffer, System.Collections.Generic.IList<System.ArraySegment<byte>> buffers, System.Net.Sockets.SocketFlags flags, byte[] socketAddress, ref int socketAddressLen, out int bytesReceived = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out System.Net.Sockets.SocketError errorCode = Success)
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory<byte> buffer, ref System.Net.Sockets.SocketFlags flags = None, byte[] socketAddress = null, ref int socketAddressLen = 0, int timeout = -1, out int bytesReceived = 0)
System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle handle, byte[] buffer, int offset, int count, System.Net.Sockets.SocketFlags socketFlags, out int bytesTransferred)
System.Net.Sockets.dll!System.Net.Sockets.Socket.Receive(byte[] buffer = {byte[14]}, int offset = 0, int size, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode = 2116193192)
System.Net.Sockets.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size)
System.Private.CoreLib.dll!System.IO.BinaryReader.ReadBytes(int count = 14)
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcHeader.TryParse(System.IO.BinaryReader reader = {System.IO.BinaryReader})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint endpoint = {Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint}, Microsoft.Diagnostics.NETCore.Client.IpcMessage message = {Microsoft.Diagnostics.NETCore.Client.IpcMessage})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo()
Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.EndpointInfo.FromProcessId(int processId = 39)
Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.ClientEndpointInfoSource.GetEndpointInfoAsync.AnonymousMethod__2()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<Microsoft.Diagnostics.Monitoring.EndpointInfo>.InnerInvoke()
System.Private.CoreLib.dll!System.Threading.Tasks.Task..cctor.AnonymousMethod__274_0(object obj)
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread = {System.Threading.Thread}, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot = Id = 2, Status = Running, Method = "Microsoft.Diagnostics.Monitoring.EndpointInfo <GetEndpointInfoAsync>b__2()", Result = "{Not yet computed}", System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Private.CoreLib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

This instance of dotnet-monitor is attempting to invoke the ProcessInfo command on process 39.

Process 39 is the dotnet run & process.

ps -aux:

jander      39  0.3  0.6 4053812 81360 pts/0   Tl   16:26   0:02 dotnet run
jander      64  0.6  0.0      0     0 pts/0    Z    16:26   0:04 [dotnet] <defunct>

This hang would be mitigated if cancellation on the DiangosticsClient was implemented: dotnet/diagnostics#2151

[jander-msft]

Killing the target process (process 39) makes dotnet-monitor return an HTTP status of 500 on the blocked requests.

All HTTP requests are blocked by this when running dotnet-monitor in 'connect' mode because all of the requests enumerate the diagnosable processes, which invokes the ProcessInfo command on each process.

[jander-msft]

The "dotnet run" process (process 39) is still running however the child process that it spawns to run the built application is zombied (the Z in the STAT column above means it exited but the parent process hasn't observed the exit, so the process block isn't cleaned up yet).

[jander-msft]

If I run it with nohup (e.g. nohup dotnet run &), then everything works correctly: the child process runs (and if a web app, responds to http requests) and dotnet-monitor is able to enumerate all of the processes.