Fix persistence liveness check deadlock #269
Conversation
| @@ -189,7 +216,7 @@ protected override void PreStart() | |||
|
|
|||
| private void ScheduleProbeRestart() | |||
| { | |||
| Context.System.Scheduler.ScheduleTellOnce(_delay, Self, CreateProbe.Instance, Self, _shutdownCancellable); | |||
| Timers.StartSingleTimer(CreateProbeTimerKey, CreateProbe.Instance, _delay); | |||
There was a problem hiding this comment.
Replace scheduler timer calls with IWithTimer to prevent leaks.
| private readonly string _id; | ||
| private readonly Cancelable _shutdownCancellable; |
There was a problem hiding this comment.
Not needed anymore because we drop the scheduler timer
| ScheduleProbeRestart(); | ||
| return true; | ||
|
|
||
| case CreateProbe: | ||
| if(_logInfo) | ||
| _log.Debug("Recreating persistence probe."); | ||
|
|
||
| Timers.StartSingleTimer(TimeoutTimerKey, CheckTimeout.Instance, _timeout); |
There was a problem hiding this comment.
Guarding against suicide actor deadlock
There was a problem hiding this comment.
We're doing it here inside the probe actor instead of inside the suicide actor because mailbox processing can be blocked inside a persistence actor, making timers unreliable.
| case CheckTimeout: | ||
| const string errMsg = "Timeout while checking persistence liveness. Recovery status is undefined."; | ||
| _log.Warning(errMsg); | ||
| _currentLivenessStatus = new PersistenceLivenessStatus(errMsg); | ||
| PublishStatusUpdates(); | ||
|
|
||
| if(_probe is not null) | ||
| Context.Stop(_probe); |
There was a problem hiding this comment.
Force stop the suicide actor if it stalls/deadlocks
There was a problem hiding this comment.
Note that we also emit a new liveness status with a timeout message to let subscriber know about the problem.
| # Defines the timeout for each liveness check operation | ||
| timeout = 3s |
There was a problem hiding this comment.
Add new setting to configure check timeout value
| ScheduleProbeRestart(); | ||
| return true; | ||
|
|
||
| case CreateProbe: | ||
| if(_logInfo) | ||
| _log.Debug("Recreating persistence probe."); | ||
|
|
||
| Timers.StartSingleTimer(TimeoutTimerKey, CheckTimeout.Instance, _timeout); |
Fix potential persistence liveness check deadlock if the suicide actor never completes due to peristence actor / storage problems.