-
Notifications
You must be signed in to change notification settings - Fork 499
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
exp/ingest/pipeline: Fix pipeline data race during shutdown (#2058)
This commit fixes data race in `exp/ingest/pipeline` that can occur when `LiveSession` (and Horizon) is shut down. It also removes `updateStats` method that was known to have a data race (see comment in that method). It is not actively used right now but was being reported by race detector. Previous code handling shutdown signal in `LiveSession` can be found below: ``` errChan := s.LedgerPipeline.Process(ledgerReader) select { case err2 := <-errChan: if err2 != nil { // Return with no errors if pipeline shutdown if err2 == pipeline.ErrShutdown { s.LedgerReporter.OnEndLedger(nil, true) return nil } if s.LedgerReporter != nil { s.LedgerReporter.OnEndLedger(err2, false) } return errors.Wrap(err2, "Ledger pipeline errored") } case <-s.standardSession.shutdown: if s.LedgerReporter != nil { s.LedgerReporter.OnEndLedger(nil, true) } s.LedgerPipeline.Shutdown() return nil } ``` The problem is when shutdown signal is received, `Resume` returns `nil` so Horizon starts it's shutdown code which calls `Rollback()` (using internal `tx` object) but at the same time pipeline is still running until the code receiving from `ctx.Done` channel is executed. It means that pipeline processors can execute transactions using `tx` transaction object in DB session. To fix this: 1. We don't `select` ingest session shutdown signal when waiting for pipeline to finish processing. 2. Instead we call `Shutdown` on pipelines inside `LiveSession.Shutdown`. 3. Then we wait/block until pipelines gracefully shutdown by calling `Pipeline.IsRunning` method. 4. Finally we `close(s.shutdown)` inside `expingest/System.Shutdown()`. So the components now shut down exactly in the following order: 1. Pipelines. 2. Session. 3. Horizon Expingest System. One comment on `-1` change in tests. When `ingestSession.Run()` returns `nil` we shouldn't continue to `ingestSession.Resume()` because `nil` value means that session ended. I updated the comment in `LiveSession` and also fixed Horizon code.
- Loading branch information
Showing
4 changed files
with
76 additions
and
74 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters