Add Postgres LISTEN/NOTIFY support #82
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bensheldon
commented
Aug 17, 2020
bensheldon
commented
Aug 17, 2020
bensheldon
commented
Aug 17, 2020
bensheldon
force-pushed
the
listen_notify
branch
7 times, most recently
from
d4bebd0 to
8b22ca6
Compare
bensheldon
force-pushed
the
listen_notify
branch
from
8b22ca6 to
6ebeee0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #54
Thread Deadlock
There is a deadlock caused by the interaction between the Scheduler and the Notifier, when the Scheduler is instantiated before the Notifier is instantiated.
Problem Identified inside of Rails with Rails 6 and Classic Audoloader: Sidekiq hangs using Rails 6 app with "classic" autoloader
The deadlock reliably happens directly when running
bundle exec good_joband the implementation can be adjusted by changing these lines: https://github.com/bensheldon/good_job/pull/82/files#diff-12faa12f1aae21d6e988253da1d9333eR22-R24`ActionDispatch::DebugLocks` output
Workaround: Instantiate the Notifier before instantiating the Scheduler.
This is written out linear, but in reality the threads seem to be interwoven:
Debugging Thread Locks / Deadlocks
Run the CLI with
rbtrace:$ RBTRACE=1 bundle exec good_job --poll_interval "30" --max_threads=1. It will pause and show the PID.Use
ActionDispatch::DebugLocksviarbtracein another terminal:References and examples:
Uses a set of tables to document a deadlock: Occasional deadlocks between Dependencies::Interlock and db adapter lock rails/rails#34310
Rails5 concurrency, with_connection and executor.wrap interaction: Rails5 concurrency, with_connection and executor.wrap interaction rails/rails#26956
Helpful description from Sidekiq about a long-running job: Sidekiq hangs when calling active record in rails 5 development environment sidekiq/sidekiq#3401 (comment)
Rails5, concurrency in action method, deadlock, docs needed including for executor/reloader (Highly referenced Rails Issue): Rails5, concurrency in action method, deadlock, docs needed including for executor/reloader rails/rails#26847