Introduce usage of redbeat celery task scheduler #393
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Elkasitu
added
the
technical
added 2 commits
December 1, 2023 17:01
This commit adds celery-redbeat as a dependency of the project and sets it as the default scheduler for celery tasks. Unlike celery-beat, redbeat uses a Redis instance for keeping track of task schedules and uses a distributed mutex lock in order to ensure that only one instance of the scheduler can actually schedule tasks. This comes with the following benefits: * The scheduler can have high availability without compromising scheduling, meaning that we can have e.g. 3 pods of the scheduler running but only one will be actively doing work at any given time, if two Availability Zones happen to go down, the remaining pod will pick up the slack. * The schedule is stored in Redis, meaning that even if there are scheduler restarts, the work to be done can be picked up immediately without delays. * Tasks can be dynamically created/modified. * Faster startup. This commit also simplifies the way that the celery-beat scripts work. Closes OSIDB-1524
Elkasitu
force-pushed
the
better-scheduler
branch
from
bac28e1
to
25f9dc1
Compare
osoukup
approved these changes
Dec 5, 2023
jsvob
reviewed
Dec 5, 2023
|
|
||
| rm -f /tmp/celery_beat.pid | ||
| exec celery -A config beat | ||
| # Reuse existing standalone version |
There was a problem hiding this comment.
Why was the healthy check removed? I don't see it in the standalone version either.
There was a problem hiding this comment.
I don't think it's necessary, the celery pods should be able to run independently of the web service as they don't depend on it.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 7, 2023
This PR fixes an error while creating a local venv due to a bug in pip. It is also a follow-up to the second commit in #393. For more info, see pypa/pip#9644.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds celery-redbeat as a dependency of the project and sets it as the default scheduler for celery tasks.
Unlike celery-beat, redbeat uses a Redis instance for keeping track of task schedules and uses a distributed mutex lock in order to ensure that only one instance of the scheduler can actually schedule tasks.
This comes with the following benefits:
This commit also simplifies the way that the celery-beat scripts work.
Closes OSIDB-1524