Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Celery worker --max-tasks-per-child for Celery 4.x. #3625

Merged
merged 4 commits into from
Mar 27, 2019

Conversation

jezdez
Copy link
Member

@jezdez jezdez commented Mar 22, 2019

What type of PR is this? (check all applicable)

  • Feature
  • Bug Fix

Description

This fixes the name of the --max-tasks-per-child parameter whose name was changed in Celery 4.x and adds a --max-memory-per-child of 1/4 of total memory.

Related Tickets & Documents

Port of mozilla#364

@jezdez jezdez added this to the v7.0.0 milestone Mar 22, 2019
@jezdez jezdez requested a review from arikfr March 22, 2019 13:16
@ghost ghost assigned jezdez Mar 22, 2019
@ghost ghost added the in progress label Mar 22, 2019
@jezdez
Copy link
Member Author

jezdez commented Mar 22, 2019

/cc @jasonthomas

bin/docker-entrypoint Outdated Show resolved Hide resolved
bin/docker-entrypoint Outdated Show resolved Hide resolved
@@ -4,9 +4,10 @@ set -e
worker() {
WORKERS_COUNT=${WORKERS_COUNT:-2}
QUEUES=${QUEUES:-queries,scheduled_queries,celery,schemas}
WORKER_EXTRA_OPTIONS=${WORKER_EXTRA_OPTIONS:}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jasonthomas Is this a legitimate shell script?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is throwing a bad substitution error. It should be WORKER_EXTRA_OPTIONS=${WORKER_EXTRA_OPTIONS:-} instead.

@jezdez jezdez requested review from arikfr and removed request for arikfr March 27, 2019 14:29
@jezdez jezdez modified the milestones: v7.0.0, Next Mar 27, 2019
Copy link
Member

@arikfr arikfr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good assuming the WORKER_EXTRA_OPTIONS thing is legit.

@arikfr arikfr merged commit 1933dee into master Mar 27, 2019
@arikfr arikfr deleted the celery-max-memory branch March 27, 2019 19:08
@arikfr
Copy link
Member

arikfr commented Mar 27, 2019

Thanks.

harveyrendell pushed a commit to pushpay/redash that referenced this pull request Nov 14, 2019
* Fix Celery worker CLI parameter name that was changed in Celery 4.x.

* Set Celery worker --max-memory-per-child to 1/4th of total system memory.

* Review fixes.

* Review fixes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants