[Issue #1391] Set the gunicorn workers based on CPU count

[chouinar]

Summary

Fixes #1391

Time to review: 3 mins

Changes proposed

Set the number of gunicorn workers to 2*cpu count + 1

Context for reviewers

We temporarily set the number of workers to exactly 2 while getting the API running initially. This shouldn't be needed anymore, and would mean we are potentially underutilitizing our API instances. This number follows the recommended values:
See: https://docs.gunicorn.org/en/stable/design.html#how-many-workers
also: https://docs.gunicorn.org/en/latest/configure.html#configuration-file

Currently, we run 2 workers always. With our current API configuration, this should change to 5. We actually log the sched_getaffinity value when the process starts up. Pulling from the prod logs, this gives (trimming extra info for brevity):

{
    "name": "src.logging.config",
    "levelname": "INFO",
    "levelno": 20,
    "pathname": "/api/src/logging/config.py",
    "funcName": "log_program_info",
    "cpu_count": 2,
    "cpu_usable": 2,
    "message": "start src: CPython 3.12.2 Linux, hostname ip-10-3-0-56.ec2.internal, pid 7, user 1001(runner)"
}

The cpu_usable metric is just the len(sched_getaffinity(0)) value: https://github.com/HHS/simpler-grants-gov/blob/main/api/src/logging/config.py#L140

Additional information

To test this, you need to adjust the docker-compose file to run the following command: ["poetry", "run", "gunicorn", "src.app:create_app()"] That'll create several workers, which run uneventfully locally, but produce a lot of logs.

[chouinar]

@jamesbursa I was curious on your opinion of how we use gunicorn compared to past projects.

Past projects we setup gunicorn in our main function, while here we tell gunicorn to run our app via a specific create_app function. This means all of the code that runs in that create_app function is run extra times for each worker. That means code like logging initialization duplicates. This doesn't seem to cause any issues at the moment, but does make the initialization a bit confusing.

[coilysiren]

makes sense to me 👍🏼 I wish I knew more about this stuff

do you think threads needs a similar treatment?

[chouinar]

makes sense to me 👍🏼 I wish I knew more about this stuff

do you think threads needs a similar treatment?

I've looked around a bit regarding how to set the number of threads, and the recommendation is largely "a few". We've set it as 4 on a few projects, and I can't recall any issues occurring. Note that threads is per-worker, and I think the scaling would be based on the types of queries you receive (more threads is better when doing a lot of DB queries and waiting around I believe?).