BROKER_URL !== CELERY_BROKER_URL

[chris-hydra]

Thanks for all the great work making this image available - posting this in case anyone else has this issue.

The README for this image doesn't make it clear that BROKER_URL is deprecated in favour of CELERY_BROKER_URL.

The Celery configuration documentation makes no mention of CELERY_BROKER_URL - all examples use BROKER_URL.

If you rename your RabbitMQ service to anything but "rabbit" in your docker-compose.yml, and try (according to the configuration docs) to set BROKER_URL accordingly, this image won't work.

I will cross-post this issue on the Celery documentation site.

[yosifkit]

BROKER_URL is the variable name chosen by upstream to use in celeryconfig.py, while CELERY_BROKER_URL is the environment variable created for this image that is used to fill BROKER_URL in the provided celeryconfig.py. The value of CELERY_BROKER_URL defaults to amqp://guest@rabbit.

[chris-hydra]

Thanks for the clarification - there still needs to be some mention that CELERY_BROKER_URL effectively overwrites any BROKER_URL setting in celeryconfig.py, when celeryconfig.py is volume-mapped from the host machine.

I would think this is a pretty common use-case - set BROKER_URL in celeryconfig.py on the host machine, and to make that configuration available to the guest Docker image via a volume mapping. The Celery Docker image breaks that, and it needs to be made clear in the Docker image docs this CELERY_BROKER_URL - celeryconfig.py interaction.

To quote immunda:

Perhaps this is more of a Celery problem, but unfortunately there's undocumented behaviour where CELERY_BROKER_URL envvar will always trump BROKER_URL as provided in a config file, as can be seen at https://github.com/celery/celery/blob/master/celery/app/utils.py#L106 which I believe results in the behaviour I encountered above.

[leimbag]

I have encountered the same problem, how to set BROKER_URL to make it valid, I want to use custom broker url, not use default CELERY_BROKER_URL,

[yosifkit]

@leimbag how are you setting BROKER_URL and running celery? If you provide your own celeryconfig.py with BROKER_URL set it will mask the file provided by the image.

$ docker run -v /path/to/my/celeryconfig.py:/home/user/celeryconfig.py [more docker args...] celery

Or if you just want to change BROKER_URL at run time, just set CELERY_BROKER_URL:

$ docker run -e CELERY_BROKER_URL=amqp://my-custom-host celery

[enanablancaynumeros]

I was accidentally using CELERY_BROKER_URL as environment variable and using broker_url in the celery_config file. Thanks to this post I found the magic that was changing silently the variable!

[tianon]

I was going through old PRs, and ended up filing #12 as a result -- if you've got time, please go check my assumptions before we deprecate this image entirely. ❤️ 🙏

[tianon]

See docker-library/docs#880 for more details of the deprecation of this image.