Make EvaP pip-installable and publish to PyPI

[niklasmohrin]

Closes #2303

Since we don't want to use nix in production (for example because security updates might take a long time to propagate through nixpkgs / through our github release cycle), pip install . or pip install evap seems to be the most canonical way. This allows us to install security patches via pip install --upgrade evap.

If we really need to override a dependency version beyond the allowed range in pyproject.toml, we can install EvaP from sources outside of PyPI. The script to build the release artifacts is available as nix run .#build-dist. It is possible to run this on a development machine and copy the .whl file over to the server.

The release wheel includes compiled TypeScript, SCSS and translation files, so there is no need to install NodeJS or gettext on the production server.

The new release workflow is:

1. Add translations and commit them.
2. Bump version in pyproject.toml, run nix develop .#impure --command uv lock, commit it, tag it, push it.
3. Create Github Release.
4. Github Actions will be triggered when the Github release is published and publish to PyPI.

The versioning scheme we agreed on is year-month-patch, for example "2024.11.0". I suppose we should take care that we don't necessarily push breaking changes like updating the Python version in a new patch.

Since upgrading to the next stable revision of EvaP is handled by pip, there is no need for the release branch anymore.

See

• https://packaging.python.org/en/latest/tutorials/packaging-projects/
• https://docs.pypi.org/trusted-publishers/using-a-publisher/

[niklasmohrin]

See https://test.pypi.org/project/evap/ and https://pypi.org/project/evap/

[niklasmohrin]

@richardebeling Thoughts? Furthermore, if you create a PyPI account, I can add you as a co-owner of the package. (I also requested the creation of an e-valuation organization on PyPI, not sure if it will be nicer / needed, we'll see once it's processed)

[niklasmohrin]

Should we add the organization members here? Right now, PyPI displays "EvaP Developers" under unverified details, I suppose we could also just add the names which will hopefully be deduplicated then

[niklasmohrin]

Looks like it won't be deduplicated, I suppose no one really cares about this, see https://test.pypi.org/project/evap/2024.10.0/ or https://pypi.org/project/black/

[richardebeling]

I must be blind -- what section exactly would contain duplicated names? I see 5 unique user names on the black project page under "Maintainers"

I'd be fine with just putting "EvaP Developers", which will always remain correct.

Do we even need this in the pyproject.toml? I would expect that for username mapping, you need to configure this on PyPI separately anyway?

[niklasmohrin]

"Ambv" is the same person as the unverified "author" (=> big projects don't care about about unverified data) and also on the evap release my name appears twice (=> it doesn't get deduplicated). Therefore, "evap developers" also seems fine to me.

We need to add something because otherwise poetry will complain :/

[richardebeling]

Ah, I see -- I was looking for duplication inside one list. Yeah, "evap developers" 👍

[niklasmohrin]

One thing we discussed in person was that we now don't really need a full checkout of the EvaP repository on production. In fact, this is a chance to move most of the (apache-)deployment files out of the main repository and into a separate repository in the e-valuation organization. @janno42 agreed that this would be a nice (while not really needed) change. The only drawback seems to be that testing the backup process in the EvaP repository will be more complicated. Currently, I believe that we would want to pin a commit of the deployment repository, for example through flake.lock, and run the tests here. I believe that we could add the backup process test logic through nix in the deployment repository and then import this test in the EvaP repository to use the updated source of EvaP. In particular, this would allow us to share the test definition and "easily" run the same tests in both repos. I am not too settled on this approach though.

Thoughts @richardebeling ?

[richardebeling]

@richardebeling Thoughts? Furthermore, if you create a PyPI account, I can add you as a co-owner of the package. (I also requested the creation of an e-valuation organization on PyPI, not sure if it will be nicer / needed, we'll see once it's processed)

While certainly unexpected, I think it's a funny and efficient solution, and I don't have any reasons against doing this.

One thing we discussed in person was that we now don't really need a full checkout of the EvaP repository on production. In fact, this is a chance to move most of the (apache-)deployment files out of the main repository and into a separate repository in the e-valuation organization. janno42 agreed that this would be a nice (while not really needed) change. The only drawback seems to be that testing the backup process in the EvaP repository will be more complicated.

Why? "Backup" essentially is "Dump database via dumpdata", isn't it? This would be separate from apache and all the other deployment aspects. I can see that the "Upgrade" part of the script wouldn't be testable -- we can compromise on that, I think. It's important that backup and especially restore is tested.

---

Re (separate repo, version pinning): Adds complexity, but if there is a need for this or it helps in any way, I guess we won't die 🤷‍♂️

A proper release procedure with some github magic action could also be used to deliver compiled TS and SCSS files automatically, something that I dislike a bit about our current approach (necessity to have npm somewhere). (We should keep in mind though: Automation is always nice up to the point where it stops working)

[niklasmohrin]

Some commands (like the one here) don't make sense when used with python -m evap outside of a development checkout. Do we care (now)?

[niklasmohrin]

Since we are already setting the variable here, it might be counterintuitive if someones does

from evap.settings import *

DATADIR = Path("...")

Do we care (now)?

[niklasmohrin]

I updated the PR description to fit the current state of things. I tested the currently published release with the following suggested setup in a podman container (podman run --rm -p 8888:80 -it ubuntu):

apt update && apt upgrade -y && apt install -y sudo redis postgresql apache2 python3 python3-venv libapache2-mod-wsgi-py3

service redis-server start
service postgresql start
service apache2 start

sudo -u postgres psql
create user evap password 'evap' createdb;
create database evap owner evap;
exit

useradd -m evap
sudo -i -u evap
mkdir evap && cd evap
python3 -m venv venv
. venv/bin/activate
pip install evap[psycopg-binary]
# set up productionsettings.py and wsgi.py with template
python -m evap collectstatic
python -m evap migrate
# collectmessages not needed, already included in release

exit # venv
exit # sudo -i
chown -R evap:www-data /home/evap

cd /etc/apache2
# set up sites-available/100-evap.conf with template, replace /evap with /home/evap/evap
a2enmod headers
a2enmod wsgi
a2dissite 000-default
a2ensite 100-evap
service apache2 restart

Afterwards, evap should be available at the given port. To log in, one should also run python -m evap reload_testdata after collectstatic.

[niklasmohrin]

We made https://github.com/e-valuation/evap-deployment as a place to move the deployment files, because production will not have a checkout of the EvaP repository anymore