Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust RQ job priorities #4301

Merged
merged 3 commits into from
Nov 6, 2019
Merged

Adjust RQ job priorities #4301

merged 3 commits into from
Nov 6, 2019

Conversation

rauchy
Copy link
Contributor

@rauchy rauchy commented Nov 4, 2019

Currently, all RQ jobs are distributed between 3 queues:

  1. periodic - handles periodic jobs stated in schedule.py.
  2. schemas - handles refresh_schemas jobs.
  3. default - for all other jobs.

A couple of issues that this causes:

  1. Since different types of jobs go to default, it's hard to get an understanding of the current distribution of jobs. A solution to this might be to split default into job-specific queues, then we could just observe the queue size.
  2. Within periodic, the refresh_queries job needs to have the highest priority.
  3. Within default, the send_email jobs needs to have the highest priority.

@rauchy rauchy requested a review from arikfr November 4, 2019 12:55
@rauchy rauchy merged commit 6f791a0 into master Nov 6, 2019
@rauchy rauchy deleted the rq-job-priorities branch November 6, 2019 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants