feat: pattern for running workers in a separate container

[peterschutt]

We spoke about how there may be times where we don't want the worker to be running in the same container as the starlite app, but rather have n dedicated containers running n workers. This could be useful when the workers are doing cpu bound things.

The basic pattern uses starlite to manage the worker startup/shutdown, so I suppose we'd just be letting SAQ do this for the standalone worker container case.

Also, it could be the same container, just with different entrypoints.. however, that would mean that the worker containers would have all of the starlite dependencies..

Maybe this gets us to the point where even starlite is an optional dependency..

[cofin]

I came to open an issue for this very thing today.

I actually implemented a pattern here that allows you to run the worker in-process or as a standalone. I'm not sure if there's a better way to handle this, but it's working in my local tests.

You can see the launcher I added here that extends the existing saqlalchemy configs.

[peterschutt]

I actually implemented a pattern here that allows you to run the worker in-process or as a standalone

Reading that is what reminded me to put this ticket in:)

We are looking for a way to just stand up the worker with access to the service objects, and not have to have starlite as a dep in that environment. This would support running the worker in a separate container to the server.

But running in multiple processes is equally valid, so will def. incorporate your pattern!