rabbitmq has known issues, replace queue backend to resolve them

[biblicabeebli]

Celery/Celery tasks are sometimes.... unsuccessful in connecting, the system emits an ampq connection error. The reason is unclear, but it appears to be a rabbitmq issue.

If this issue remains swap the system over to use reddis for the celery task queue backend.

[biblicabeebli]

Newer versions of celery have a django backend! that would reduce complexity of deploying and of security group management potentially substantially!

needs research.

[biblicabeebli]

Misread documentation about this, django cannot be used for the queue, our best option is to replace with redis and hope it isn't as problematic.

[biblicabeebli]

alternative: try out Huey, a less complex task dispatch python library.
https://huey.readthedocs.io/en/latest/

[biblicabeebli]

Okay, last straw, we managed to get two concurrent data processing tasks on the same user. this can only happen if rabbitmq is either not atomic, or if rabbitmq does its whole totally-unnacceptable silent dropping of api calls thing.

Use of rabbitmq is now considered a bug, we are swapping to reddis as soon as I have dev time for it.

[biblicabeebli]

• rabbitmq has known issues, replace queue backend to resolve them #93
• General improvements to automation of data analyses, data processing, and other background tasks #305
• Update Celery [finally] #353
• Django 3.2 lts support ends on 1 Apr 2024 #352
• Ubuntu 18.04 ~end of life on aws ... for new accounts only? What? #331 - ubuntu platform upgrade - required before celery and rabbitmq.

[hackdna]

I recommend watching this talk before committing to replacement of RabbitMQ: https://2023.djangocon.us/talks/mixing-reliability-with-celery-for-delicious-async-tasks/

[biblicabeebli]

aaaaand celery 5.4 has redis bugs. https://github.com/celery/celery/milestone/40

[biblicabeebli]

deploy now uses updated ubuntu, python version, but something changed in rabbitmq configuration so I need to work that out.

[biblicabeebli]

keeping this open so that we update to 5.4 along with ubuntu 24.04....

[biblicabeebli]

Celery 5.4 is released, dogfooding on staging server (I've read release notes, no anticipated issues).

[biblicabeebli]

The original reason for which I opened this is actually more frequent after a Sentry update. Fantastic.

[biblicabeebli]

putting this issue on pause because I don't even know if redis will fix this.

[biblicabeebli]

closing issue, there is no need to change from rabbitmq at this time even with the problems it seems to cause as there's no evidence that these are not actually just celery problems.