log_eviction_stm: fix shutdown hang with full queue

Scenario:

* delete-records enters the queue for offset 10
* 3 more storage events enter the queue (queue is now full)
* delete-records attempts to enter the queue for offset 15
* shutdown begins
* consensus gate is closed. other persisted_stms can no longer make readers, can’t move forward their max collectible offset
* eviction event 10 repeats indefinitely in the event processing loop
* log_eviction_stm is stuck in apply trying to enqueue
* consensus cannot be stopped because a reader is kept open as we stuck in apply

Solution:

A temporary, not pretty solution, is in this commit which tries to
determine if raft is in the process of shutting down, in which case, the
eviction stm does so too before it is officially asked to stop.

Signed-off-by: Noah Watkins <[email protected]>

src/v/cluster/log_eviction_stm.cc

@@ -77,6 +77,12 @@ ss::future<> log_eviction_stm::handle_log_eviction_events() {
     auto gh = _gate.hold();
 
     while (!_as.abort_requested()) {
+        if (_raft->stopped()) {
+            _queue_mutex.broken();
+            _queue.abort(
+              std::make_exception_ptr(ss::abort_requested_exception()));
+            break;
+        }
         /// This background fiber can be woken-up via apply() when special
         /// batches are processed or by the storage layer when local
         /// eviction is triggered.

src/v/raft/consensus.h

@@ -451,6 +451,8 @@ class consensus {
 
     model::offset get_flushed_offset() const { return _flushed_offset; }
 
+    bool stopped() const { return _bg.is_closed(); }
+
 private:
     friend replicate_entries_stm;
     friend vote_stm;