Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery worker, Celery beat fixes for AfS (dev/1.0.x) Docker #9797

Closed
ekansa opened this issue Jul 18, 2023 · 2 comments
Closed

Celery worker, Celery beat fixes for AfS (dev/1.0.x) Docker #9797

ekansa opened this issue Jul 18, 2023 · 2 comments
Assignees

Comments

@ekansa
Copy link

ekansa commented Jul 18, 2023

I was trying to test the Bulk Data Manager in AfS, but I had trouble getting the celery worker working in my Docker set up. The superviord.log inside the afs7 container gave some nonspecific, unhelpful logs:

root@ae93defc8769:/web_root/afs# cat /var/log/supervisor/supervisord.log
2023-07-18 18:25:10,270 INFO Included extra file "/web_root/afs/docker/./conf.d/afs-celeryd.conf" during parsing
2023-07-18 18:25:10,270 INFO Set uid to user 0 succeeded
2023-07-18 18:25:10,273 INFO RPC interface 'supervisor' initialized
2023-07-18 18:25:10,273 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-07-18 18:25:10,274 INFO daemonizing the supervisord process
2023-07-18 18:25:10,275 INFO supervisord started with pid 14
2023-07-18 18:25:11,277 INFO spawned: 'celery' with pid 27
2023-07-18 18:25:11,512 INFO exited: celery (exit status 1; not expected)
2023-07-18 18:25:12,515 INFO spawned: 'celery' with pid 43
2023-07-18 18:25:12,729 INFO exited: celery (exit status 1; not expected)
2023-07-18 18:25:14,733 INFO spawned: 'celery' with pid 90
2023-07-18 18:25:15,121 INFO exited: celery (exit status 1; not expected)
2023-07-18 18:25:18,125 INFO spawned: 'celery' with pid 117
2023-07-18 18:25:18,352 INFO exited: celery (exit status 1; not expected)
2023-07-18 18:25:19,354 INFO gave up: celery entered FATAL state, too many start retries too quickly

If one entered the afs container and tried to manually invoke the celery worker with:
python3 -m celery -A afs.celery worker --loglevel=INFO

celery gave a more helpful error message:
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key UTC'

Stack exchange suggested updating some timezone related Python packages to resolve.

So we may want to add to the pip installs in the Dockerfile for AfS
pip install pytz --upgrade && pip install tzdata --upgrade

This update seems to make Celery (and supervisord) happy. I tested supervisord/celery with both Rabbit_MQ and Redis as brokers, and both seem to work with this fix.

As @chiatt mentioned, we should test to make sure these updated libraries (esp. pytz) don't break any of the existing Arches dependencies.

@ekansa ekansa changed the title Celery worker, Celery Beat fixes for AfS (dev/1.0.x) Docker Celery worker, Celery beat fixes for AfS (dev/1.0.x) Docker Jul 18, 2023
@chiatt
Copy link
Member

chiatt commented Jul 20, 2023

Moving this issue to core because it will likely affect other projects

@chiatt chiatt transferred this issue from archesproject/arches-for-science Jul 20, 2023
@chiatt chiatt added this to pipeline Jul 20, 2023
@chiatt chiatt self-assigned this Jul 26, 2023
@chiatt
Copy link
Member

chiatt commented Jul 26, 2023

re #9800

@chiatt chiatt moved this to 👀 In Review in pipeline Jul 26, 2023
@chiatt chiatt moved this to 👀 In review in afs-pipeline Jul 26, 2023
njkim added a commit that referenced this issue Jul 27, 2023
…ctions

Ensure pytz and tzdata are at version compatible with Celery, re #9797
@chiatt chiatt closed this as completed Jul 28, 2023
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in afs-pipeline Jul 28, 2023
@github-project-automation github-project-automation bot moved this from 👀 In Review to ✅ Done in pipeline Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Archived in project
Development

No branches or pull requests

2 participants