You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.
The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence ExtendLockLifetime is called more.
https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314 ExtendLockLifetime is called by Timer, and Timer callbacks are dispatched to ThreadPool.
ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of ExtendLockLifetime invocations, thus it causes ASP.NET server to almost stall.
Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.
The text was updated successfully, but these errors were encountered:
We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.
The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence
ExtendLockLifetime
is called more.https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetime
is called byTimer
, and Timer callbacks are dispatched to ThreadPool.ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of
ExtendLockLifetime
invocations, thus it causes ASP.NET server to almost stall..NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading
Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.
The text was updated successfully, but these errors were encountered: