System.AccessViolationException -- seems like yarp has some issues with memory

[asafdl]

Describe the bug

This error pops up in production about 5 times a day, couldn't connect it to anything special, the pod doesn't go OOM, yarp doesn't report any special spikes in requests or latency, and no other errors in logs connected to this.

each yarp deployment is handling an avg. of 4 requests per second which shouldn't be too many i think

To Reproduce

Not sure

Got Exceptions? Include both the message and the stack trace

"Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Buffer.Memmove(Byte ByRef, Byte ByRef, UIntPtr)
   at System.String.Ctor(Char*, Int32, Int32)
   at System.Diagnostics.Tracing.EventSource.DecodeObjects(System.Object[], System.Type[], EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteToAllListeners(System.Diagnostics.Tracing.EventWrittenEventArgs, Int32, EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteEventWithRelatedActivityIdCore(Int32, System.Guid*, Int32, EventData*)
   at System.Net.Http.HttpTelemetry.WriteEvent(Int32, Byte, Byte, Int64, System.String, System.String, Int32, System.String)
   at System.Net.Http.HttpConnectionBase..ctor(System.Net.Http.HttpConnectionPool, System.Net.IPEndPoint)
   at System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ConstructHttp11ConnectionAsync>d__108 ByRef)
   at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.HttpConnectionPool+<ConnectToTcpHostAsync>d__104.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<ConnectToTcpHostAsync>d__104, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Net.Sockets.Socket+<<ConnectAsync>g__WaitForConnectWithCancellation|285_0>d.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Sockets.Socket+<<ConnectAsync>g__WaitForConnectWithCancellation|285_0>d, System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Net.Sockets.SocketAsyncEventArgs+<<DnsConnectAsync>g__Core|112_0>d.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Sockets.SocketAsyncEventArgs+<<DnsConnectAsync>g__Core|112_0>d, System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()"

Further technical details

• Include the version of the packages you are using 2.1.0
• The platform (Linux/macOS/Windows) linux - k8s

[MihaZupan]

This appears to be an issue with the built-in telemetry in .NET, not specific to YARP.
Looking at the code, nothing stands out to me as wrong.

Is the stack trace always the same?
Can you capture a memory dump of the process when such a crash occurs?

[asafdl]

the stack trace is always the same, since this error happens in prod and is sporadic not sure how i can capture a mem dump on it

[MihaZupan]

You can collect a dump when crashing by setting an environment variable, see https://learn.microsoft.com/dotnet/core/diagnostics/collect-dumps-crash

[asafdl]

it doesn't seem to crash yarp, maybe kestral or whatever built in lib catches this error

[MihaZupan]

I believe that seeing Fatal error. indicates that the process will be terminated.
Can you check if you're seeing the AV exception under FirstChanceException, and whether you can force a crash there (Environment.FailFast)?

[karelz]

Triage: Not actionable without a dump. Once you have a dump, feel free to reopen.

Note: Given that the stack is suspicious, check that it happens on more than 1 machine. Chasing down 1 machine config/HW issues is very time consuming for everyone.

[MihaZupan]

You were hitting dotnet/runtime#99603. Please watch that issue for updates