Docker Deployment for Kelvin core

[JersyJ]

Issue

Part of #503.

Summary

Introduces ability to deploy the Kelvin application (except Django RQ workers) inside Docker with docker-compose.yml (profile "prod"). The deployment includes the following networked services:

• The Django backend
• nginx, which will serve the Django backend
• Postgres (DB)
• Redis (cache)

Added pre-commit
Introduced additional variables for deployment configuration
Created Dockerfile for the core app
Refactored docker-compose.yml for local development and production deployment with profile "prod" (docker compose --profile prod up)
Created asgi.py for ASGI web servers
Added CSRF trusted domains for reverse proxy (NGINX) usage
Included example NGINX configuration

Testing

Needs Proper Testing! I have tested the basics. Someone with more experience should conduct thorough tests.

Nginx

To run Nginx, you need to place privkey.pem and fullchain.pem in the appropriate folder. For testing purposes, you can generate them via openssl :
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ -keyout ./privkey.pem \ -out ./fullchain.pem \ -subj "/C=US/ST=YourState/L=YourCity/O=YourOrganization/CN=localhost".

Remaining Tasks:

• The setup of Django RQ workers is missing

[Kobzol]

Thanks, this looks great! Left a few comments inline.

Maybe to clarify a bit, I think that we don't want to commit too many specific things about the deployment into git (e.g. the nginx config). We have some configuration files on the existing server and we'll likely port them over manually, and these will differ on each specific production deployment anyway. The goal is to have Dockerfile + docker-compose.yml prepared in a way so that it can be deployed easily, and that it can be configured, i.e. I can pass it nginx config/certificates/log dir, local_settings.py, paths where Postgre and Redis data should be stored etc.

[Kobzol]

Uhh, I only reviewed the first commit, sorry :) I'll take a look at the rest.

[JersyJ]

Added pre-commit to avoid future CI issues before pushing to remote.

[Kobzol]

Ok now I saw all the commits and it makes more sense 😆 Sorry.

In general, this already has pretty much almost everything that I wanted, so that is awesome :) It would be nice to de-duplicate the docker-compose files and maybe simplify the Dockerfile a bit (it seems a bit overkill to me at the moment), but otherwise this looks really good.

I hope that the asgi changes won't affect the normal wsgi operation on the current production deployment, especially since the semester starts in two days 😆

[Kobzol]

I tried to deploy the website locally using Docker compose and it actually works!! ❤️ That's really awesome 😆

Left a few more comments.

When I build the image locally, it prints this:

useradd warning: uvicorn's uid 1001 is greater than SYS_UID_MAX 999

Maybe we should use an UID below 1000, just to be sure?

[JersyJ]

Maybe we should use an UID below 1000, just to be sure?

True, yes we should. Changed.

Also changed to WSGI, since I looked through the views and it would be better for them to have WSGI.

Also moved the ruff.toml to pyproject.toml, to get rid of files in the root of the project.

[Kobzol]

Thank you very much for working on this :) This looks really great now. Since it's the first week of the semester, I will delay merging until the weekend, because this also has some changes that could in theory affect the production deployment, which is.. brittle, to say the least :)

[Kobzol]

We had some issues with Kelvin over the weekend and had to do a bunch of reverts, so I didn't merge this yet. I'll try it once Kelvin has a bit less traffic.

[geordi]

We can also "test deploy" it on the development server. Just saying.

[JersyJ]

If you want some help setting up a proper deployment, feel free to reach out to me.

[Kobzol]

I took the liberty of re-creating the requirements.txt file and removing the asgi.py file, which should not be needed, to reduce the change of causing some issues on production. Let's see if everything still works after this is merged :)

[Kobzol]

Thank you very much!

[Kobzol]

Why the heck does pre-commit have all these dependencies.. sigh.

[JersyJ]

Pre-commit is powerful tool and is designed to handle much wider range of tools and use-cases. I added it as a dev dependency, so it shouldn’t be an issue.

[Kobzol]

We currently install deps using uv pip sync because of some legacy reasons, so it installs everything 😅 But it doesn't really matter.

[JersyJ]

In my Dockerfile I added parameter which excludes them. So after you switch , no more dev dependencies 😅

[Kobzol]

Looks like everything went through, yay 🎉 The next step is for someone to SSH to the new server and try to deploy Kelvin there using Docker.. :)

[JersyJ]

I took the liberty of re-creating the requirements.txt file and removing the asgi.py file, which should not be needed, to reduce the change of causing some issues on production. Let's see if everything still works after this is merged :)

Okk, maybe the next step would be to get rid of the requirements.txt after switching to Docker deployment :)

[Kobzol]

Yeah, of course :) requirements.txt is there only as an artifact to keep compatibility with our old virtualenv, which I did not want to recreate. Once we use uv full and/or switch to a Dockerized deployment, we should nuke it.