Merge pull request #306 from AKVorrat/finalize-4.0

Finalize 4.0

.github/workflows/build.yaml

@@ -155,6 +155,27 @@ jobs:
           cd dist
           find . -type f -exec curl --netrc --upload-file '{}' "$NEXTCLOUD_BASE" \;
 
+  pypi-publish:
+    name: Upload release to PyPI
+    needs: build
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/dearmep
+    permissions:
+      id-token: write
+    steps:
+      - name: Retrieve Python package
+        uses: actions/download-artifact@v3
+        with:
+          name: python-package
+          path: dist
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://pypi.org/legacy/
+
   deploy:
     needs: build  # Technically, it only needs "backend", but we want the build to succeed first.
 

README.md

@@ -96,6 +96,19 @@ Get started quickly and explore all aspects of this project with our detailed do
 
 ---
 
+## 🆘 Support
+
+If you need help setting up DearMEP, have questions, or found a bug, there are several ways to contact us:
+
+- [GitHub Discussions](https://github.com/AKVorrat/dearmep/discussions) for general Q & A.
+- [GitHub Issues](https://github.com/AKVorrat/dearmep/issues) for bugs and contributions.
+- Our [public chat](https://signal.group/#CjQKIIvA-iVKQh2KW5Y5Ng8jaYsgd9ScDmQkUkJLuZ1mSG3yEhDUzv3WmAAVHkm6d-cLpy50) on [Signal](https://signal.org/).
+
+Some of the people behind DearMEP are also available for paid consulting services and feature development.
+Contact us and let us know what you need.
+
+---
+
 ## 🛡️ License
 
 This project is licensed under the **AGPL License**. See the [`LICENSE.md`](LICENSE.md) file for details.

doc/data-conversion.md

@@ -286,7 +286,7 @@ In fact, the example config should have this setting point to a file called `dea
 So, let’s ask DearMEP to set up a fresh database.
 
 ```console
-$ alembic upgrade head
+$ dearmep alembic upgrade head
 The configuration file was not found. This usually means that you did not set the DEARMEP_CONFIG environment variable to the config file name, or its path is incorrect.
 Traceback (most recent call last):
 …
@@ -296,7 +296,7 @@ That didn’t work.
 See, you not only need a configuration file, you also need to _tell_ DearMEP where to _find_ it.
 By default, it will look for a file named `config.yaml` in the current directory.
 Since we named ours differently (yes, of course this was for demonstration purposes), we need to set the environment variable `DEARMEP_CONFIG` to the path to this file.
-This can be done by prefixing the `dearmep` command with it, e.g. `DEARMEP_CONFIG=dearmep-config.yaml alembic upgrade head`, but you’d need to do it every time you invoke DearMEP.
+This can be done by prefixing the `dearmep` command with it, e.g. `DEARMEP_CONFIG=dearmep-config.yaml dearmep alembic upgrade head`, but you’d need to do it every time you invoke DearMEP.
 
 The easier way is probably to execute `export DEARMEP_CONFIG="$(pwd)/dearmep-config.yaml"` once.
 This setting will persist while you’re logged in to your current shell.
@@ -306,7 +306,7 @@ For this, create a file called `.env` and write `DEARMEP_CONFIG=dearmep-config.y
 Now, initializing the database should work:
 
 ```console
-$ alembic upgrade head # no output means nothing went wrong
+$ dearmep alembic upgrade head  # no output means nothing went wrong
 $ ls -lh dearmep.sqlite
 -rw-r--r-- 1 scy scy 64K Jul  8 14:13 dearmep.sqlite
 ```

server/README.md

@@ -7,7 +7,7 @@ SPDX-License-Identifier: AGPL-3.0-or-later
 
 # DearMEP Server
 
-This is the server side of DearMEP.
+This is the server side of [DearMEP](https://github.com/AKVorrat/dearmep).
 It provides a RESTful API and employs
 
 * [FastAPI](https://fastapi.tiangolo.com/) as its application framework,
@@ -21,21 +21,22 @@ as well as several other libraries to do its job.
 
 * [Pydantic](https://docs.pydantic.dev/1.10/)-powered rigorous validation of user-supplied data
 * Fully configurable rate limiting and telephony subsystem to protect against malicious users and manage spending
-* [Prometheus metrics](../doc/metrics.md) to observe the system's health
+* [Prometheus metrics](https://github.com/AKVorrat/dearmep/blob/main/doc/metrics.md) to observe the system's health
 * Convenient command line interface and composable import and export tools
 * Privacy-first design, keeping information stored about users to a minimum
 
 ## Installation
 
-DearMEP is not yet available on PyPI.
-For now, please install it "for development", as described below.
+DearMEP is available [on PyPI](https://pypi.org/project/DearMEP/) and can be installed and run using normal Python packaging tools, e.g. `pip install dearmep` or `uvx dearmep`.
+
+The packages include a matching version of the [client](https://github.com/AKVorrat/dearmep/blob/main/client/README.md) and are able to serve it via HTTP, see [serving static files](#serving-static-files-eg-the-client) below.
 
 ## Installation for Development
 
 1. Install [uv](https://docs.astral.sh/uv/).
 2. Clone this repository and `cd` into it.
 3. Run `uv sync --all-extras`.
-  * See below on which extras this will install. Alternatively, you can select individual extras with `--extra NAME` or don’t install any extras at all.
+    * See below on which extras this will install. Alternatively, you can select individual extras with `--extra NAME` or don’t install any extras at all.
 4. Done.
 
 uv takes care of managing a [virtual environment](https://docs.python.org/3/tutorial/venv.html) in `.venv/` for you, containing DearMEP and all of its dependencies, without messing with your global installation.
@@ -68,29 +69,31 @@ The most accurate documentation about the command line interface is always what
 * `convert`: Several tools to convert data into into formats that DearMEP can use, or intermediate formats for further manipulation.
 * `import`: Import Destinations (people to contact, e.g. MEPs) or scoring information into the database.
 * `db`: Check the database for issues, add files to the blob storage.
+* `alembic`: Launch a pre-configured version of the [Alembic](https://alembic.sqlalchemy.org/) CLI, i.e. with `ALEMBIC_CONFIG` already pointing to [our configuration](https://github.com/AKVorrat/dearmep/blob/main/server/dearmep/migrations/alembic.ini).
 * `dump`: Print example configuration files and specifications.
 * `version`: Print the version numbers of Python, DearMEP, and its most important libraries.
 * `check`: Run checks against your configuration, e.g. whether all strings are translated in all languages.
 
 ## Providing a Configuration File
 
 Please set the environment variable `DEARMEP_CONFIG` (which defaults to `config.yaml`) to the name of a YAML file containing the configuration.
-See [`example-config.yaml`](dearmep/example-config.yaml) for an extensively commented example.
+See [`example-config.yaml`](https://github.com/AKVorrat/dearmep/blob/main/server/dearmep/example-config.yaml) for an extensively commented example.
 
 You can use `dearmep dump example-config` to produce a config file, which you can then modify according to your needs.
+**Make sure to at least change the secrets, i.e. `authentication.secrets.pepper` and `authentication.secrets.jwt.key`.**
 
 ## Providing MEPs & Scoring
 
 DearMEP does not come with a built-in list of Members of Parliament.
 That list would change quite often, and you might only want to offer your users a subset of them, or an entirely different group of people and not MEPs at all.
-(For that reason, we often call them _Destinations_ instead, i.e. people that can be called. For more on some of our special terms, please see the [glossary](../doc/glossary.md).)
+(For that reason, we often call them _Destinations_ instead, i.e. people that can be called. For more on some of our special terms, please see the [glossary](https://github.com/AKVorrat/dearmep/blob/main/doc/glossary.md).)
 
 Also, one of DearMEP's key features is that you can configure, for each Destination, how important it is to contact them.
 Usually, you'd want to focus on MEPs that are still undecided on the policy your campaign attempts to influence, and not waste time and money calling people who are guaranteed to oppose you (or guaranteed to support you).
-Therefore, DearMEP uses a concept we call [Swayability](../doc/selecting-destinations.md) to calculate a kind of scoring to determine the priority assigned to a Destination.
+Therefore, DearMEP uses a concept we call [Swayability](https://github.com/AKVorrat/dearmep/blob/main/doc/selecting-destinations.md) to calculate a kind of scoring to determine the priority assigned to a Destination.
 Of course, these criteria vary wildly between the policy issue at hand, and therefore you need to provide DearMEP with your desired scores as well.
 
-We have a separate document on [how to convert data for use in DearMEP](../doc/data-conversion.md) that talks about tools and methods in general, but also how to get a list of MEPs from publicly available data.
+We have a separate document on [how to convert data for use in DearMEP](https://github.com/AKVorrat/dearmep/blob/main/doc/data-conversion.md) that talks about tools and methods in general, but also how to get a list of MEPs from publicly available data.
 
 Importing Swayability scores is done using the `dearmep import swayability` command.
 Calling it with `--help` provides information about the CSV format it expects.
@@ -101,18 +104,18 @@ The scoring algorithm is explained and configured in DearMEP's config file; look
 When DearMEP calls a User, they are presented with a pretty standard audio-based [IVR](https://en.wikipedia.org/wiki/Interactive_voice_response) menu.
 Of course the menu needs audio files to play.
 
-We have a separate document explaining [how the menu works and which files to supply](../doc/ivr.md), and we provide prerecorded IVR audio in several languages in the [DearMEP media repository](https://github.com/AKVorrat/dearmep-media/).
+We have a separate document explaining [how the menu works and which files to supply](https://github.com/AKVorrat/dearmep/blob/main/doc/ivr.md), and we provide prerecorded IVR audio in several languages in the [DearMEP media repository](https://github.com/AKVorrat/dearmep-media/).
 However, you will still need to record [at least a few messages](https://github.com/AKVorrat/dearmep-media/blob/main/README.md#voice-menu-audio) to customize them to your own Campaign.
 Also, while the media repo provides the files as `.wav` recordings for maximum fidelity and lossless editing, DearMEP expects them in Ogg Vorbis format.
-See the data conversion document's section on [converting the audio files](../doc/data-conversion.md#converting-the-audio-files) for details.
+See the data conversion document's section on [converting the audio files](https://github.com/AKVorrat/dearmep/blob/main/doc/data-conversion.md#converting-the-audio-files) for details.
 
 ## Database Migrations
 
 DearMEP is using [Alembic](https://alembic.sqlalchemy.org/) to manage the database and perform version upgrades to it.
 Currently, this is a manual process.
 
 Note that Alembic requires a configuration file telling it where to find the database and migrations.
-Fortunately, DearMEP comes with a [prepared `alembic.ini`](dearmep/migrations/alembic.ini) as well as a special CLI subcommand, `dearmep alembic`, that will invoke Alembic with the environment variable `ALEMBIC_CONFIG` already pointing to the location of this configuration file.
+Fortunately, DearMEP comes with a [prepared `alembic.ini`](https://github.com/AKVorrat/dearmep/blob/main/server/dearmep/migrations/alembic.ini) as well as a special CLI subcommand, `dearmep alembic`, that will invoke Alembic with the environment variable `ALEMBIC_CONFIG` already pointing to the location of this configuration file.
 
 ### For Administrators of a DearMEP Instance
 
@@ -147,7 +150,7 @@ See `dearmep serve --help` for other options, including how to start it via a cu
 
 ## Accessing Development Tools
 
-You can use the [`Makefile`](Makefile) to access recommended development commands easily.
+You can use the [`Makefile`](https://github.com/AKVorrat/dearmep/blob/main/server/Makefile) to access recommended development commands easily.
 
 * `make sync` to sync your environment, install dependencies, etc. This will run `uv sync --all-extras`.
 * `make fmt` runs the [Ruff](https://docs.astral.sh/ruff/) formatter on the code.
@@ -221,6 +224,8 @@ Additional available variables are:
 The generated HTML is cached in memory.
 If you modify the document or the Jinja template after its initial render, these modifications will not show up in the HTML output until DearMEP is restarted.
 
+The [`example-markdown` directory](https://github.com/AKVorrat/dearmep/blob/main/server/example-markdown) contains, as the name suggests, an example setup.
+
 ## Retrieving the OpenAPI specification
 
 A running DearMEP server will provide its OpenAPI specification at `/openapi.json` and GUIs for it at `/docs` and `/redoc`.
@@ -240,7 +245,7 @@ For nginx, and if you’re using `dearmep serve`, check out [Uvicorn’s best pr
 ## Prometheus Metrics
 
 DearMEP exports [Prometheus](https://prometheus.io/) metrics on the usual `/metrics` endpoint, providing you with insight about the health of the system as well as statistics.
-The [available metrics](../doc/metrics.md) are documented on a separate page.
+The [available metrics](https://github.com/AKVorrat/dearmep/blob/main/doc/metrics.md) are documented on a separate page.
 
 Note that you probably don't want to allow public access to these metrics.
 They don't contain user data, but do expose information about the phone numbers DearMEP itself is using to call people, which destination is being called how often and for how long, as well as cost statistics.

server/dearmep/cli/__init__.py

@@ -114,7 +114,7 @@ def run() -> None:
     # used. This is because it needs to pass the rest of argv to Alembic
     # verbatim, but argparse can't ignore arguments that look like flags.
     # <https://github.com/python/cpython/issues/61252>
-    if argv[1] == "alembic":
+    if len(argv) > 1 and argv[1] == "alembic":
         argv.insert(2, "--")
 
     args = parser.parse_args()

server/dearmep/logging.yaml

@@ -20,9 +20,11 @@ handlers:
     formatter: uvicorn
 
 loggers:
+  markdown-it:
+    level: WARNING  # DEBUG is too verbose
   sqlalchemy.engine:
     level: WARNING  # set to INFO to log queries; DEBUG to also log results
 
 root:
-  level: DEBUG
+  level: INFO
   handlers: [console]

server/pyproject.toml

@@ -11,9 +11,29 @@ description = "DearMEP helps people contact their representatives."
 requires-python = ">=3.9,<4"
 authors = [
 	{name = "Tim Weber"},
+	{name = "Tobias Mühlberger"},
+	{name = "iameru"},
+	{name = "Philipp Aaron Becker"},
+	{name = "Jörn Bethune"},
 ]
+maintainers = [
+	{name = "Tim Weber"},
+]
+readme = "README.md"
 classifiers = [
+	"Development Status :: 5 - Production/Stable",
+	"Environment :: Console",
+	"Environment :: Web Environment",
+	"Framework :: FastAPI",
 	"License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
+	"Operating System :: POSIX",
+	"Programming Language :: Python :: 3 :: Only",
+	"Programming Language :: Python :: 3.9",
+	"Programming Language :: Python :: 3.10",
+	"Programming Language :: Python :: 3.11",
+	"Programming Language :: Python :: 3.12",
+	"Programming Language :: Python :: 3.13",
+	"Topic :: Communications :: Telephony",
 ]
 dependencies = [
 	"alembic~=1.12.1",
@@ -44,6 +64,12 @@ dependencies = [
 	"pyjwt>=2.10.1",
 ]
 
+[project.urls]
+homepage = "https://dearmep.eu/"
+source = "https://github.com/AKVorrat/dearmep"
+documentation = "https://github.com/AKVorrat/dearmep/blob/main/README.md"
+issues = "https://github.com/AKVorrat/dearmep/issues"
+
 [dependency-groups]
 dev = [
 	"mypy~=1.13.0",