Multiprocess RQ workers (using supervisor)

[rauchy]

What type of PR is this? (check all applicable)

• Refactor

Description

This is an alternative implementation to the one suggested in #4233. Here we use supervisor instead of Honcho in order to spin up, monitor and restart multiple worker processes.

supervisor is installed in the Docker image and is launched inside the worker entrypoint. We use the non-daemonized mode of supervisor in order to allow the container to exit in case the supervisor process exits.

Related Tickets & Documents

#4233

[arikfr]

We still some healthcheck, right? Can Supervisor handle it?

[arikfr]

Is there a reason you didn't configure supervisord to write to stdout directly?

[rauchy]

That's even simpler :) didn't think of that! 1fb1a7e handles this.

[arikfr]

Need to update this to stdout as well.

[rauchy]

fdbbd13

[rauchy]

We still some healthcheck, right? Can Supervisor handle it?

In 092eb6c I've used supervisor_checks to run a check every minute (we should probably change the interval) - the check runs for every monitored process and asserts that the worker is behaving as expected (on top of what you get from supervisor, which is basically testing if the process is up). If a specific worker process is misbehaving, it is replaced.

We can also add more checks for CPU / memory (provided OOTB by supervisor_checks)

Let me know what you think of this.

[rauchy]

@arikfr can we merge this?