Add docker compose based testing

Our release process documentation recommends getting passing tox
results on as many popular linux distributions as time allows. Doing
this by hand is cumbersome, redundant, and error prone.

Add a directory with a helper script that automates the entire process
using docker compose and document its use in MAINTAINERS.rst.

Signed-off-by: Martí Bolívar <[email protected]>
Signed-off-by: Pieter De Gendt <[email protected]>

.gitignore

@@ -12,3 +12,4 @@ shippable/
 htmlcov/
 .dir-locals.el
 .venv/
+docker-testing/outdir

MAINTAINERS.rst

@@ -11,20 +11,35 @@ Pre-release test plan
 
      git checkout vX.Y-branch
 
-1. Make tox happy on the following first-party platforms:
+1. Make tox happy on the following first-party non-Linux platforms:
 
    - Windows 10
    - the latest macOS
-   - the latest Ubuntu LTS
 
-2. Make tox happy on other popular Linux distributions as resources allow.
-   Doing this in a container is fine.
+   Do this by hand and check for any anomalous warnings in the output.
+   Do not just trust CI.
+
+2. Make tox happy on other popular Linux distributions:
 
    - Arch
-   - the latest Ubuntu release (if different than the latest LTS)
-   - Debian stable (if its Python 3 is still supported)
+   - the latest Ubuntu LTS release
+   - the latest Ubuntu development release
+   - Debian stable
    - Debian testing
-   - Fedora
+   - the latest Fedora release
+   - the latest Fedora rawhide release
+
+   Automated infrastructure for doing this in docker is in the docker-testing
+   directory. Start by updating the Dockerfiles and compose.yaml in that
+   directory if any newer distribution versions should be tested.
+
+   Then, install docker compose in your host Linux environment and run::
+
+     cd docker-testing
+     ./run-tests.sh
+
+   Make sure to check the tox.log files mentioned in the output for any
+   anomalous warnings.
 
 3. Build alpha N (N=1 to start, then N=2 if you need more commits, etc.) and
    upload to pypi. See "Building and uploading the release wheels" below for

docker-testing/README.rst

@@ -0,0 +1,7 @@
+Docker based testing
+--------------------
+
+This directory contains helper files used for running west's tests in Docker on
+various Linux runtimes. It was originally developed for release testing.
+
+Run "./run-tests.sh" in this directory to run the tests.

docker-testing/arch/Dockerfile

@@ -0,0 +1,13 @@
+ARG TARGET
+
+FROM ${TARGET}
+
+ARG TARGET
+ENV WEST_TARGET=${TARGET}
+
+RUN pacman -Syu --noconfirm \
+           git \
+           python-pip \
+    && pacman -Scc --noconfirm
+
+RUN pip3 install --break-system-packages tox

docker-testing/compose.yaml

@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: Apache-2.0
+
+x-common: &common
+  user: ${MY_UID}:${MY_GID}
+  volumes:
+    - /etc/passwd:/etc/passwd:ro
+    - /etc/group:/etc/group:ro
+    - ..:/west
+  command: /west/docker-testing/in-container-test.sh
+  environment:
+    WEST_TOX_OUT: /west/docker-testing/outdir
+    WEST_TOX_OUT_IN_HOST: ${WEST_IN_HOST}/docker-testing/outdir
+
+services:
+  west-archlinux-latest:
+    <<: *common
+    build:
+      args:
+        TARGET: archlinux:latest
+      dockerfile: arch/Dockerfile
+
+  west-debian-stable:
+    <<: *common
+    build:
+      args:
+        TARGET: debian:stable
+      dockerfile: debian/Dockerfile
+
+  west-debian-testing:
+    <<: *common
+    build:
+      args:
+        TARGET: debian:testing
+      dockerfile: debian/Dockerfile
+
+  west-fedora-latest:
+    <<: *common
+    build:
+      args:
+        TARGET: fedora:latest
+      dockerfile: fedora/Dockerfile
+
+  west-fedora-rawhide:
+    <<: *common
+    build:
+      args:
+        TARGET: fedora:rawhide
+      dockerfile: fedora/Dockerfile
+
+  west-ubuntu-latest:
+    <<: *common
+    build:
+      args:
+        TARGET: ubuntu:latest
+      dockerfile: debian/Dockerfile
+
+  west-ubuntu-devel:
+    <<: *common
+    build:
+      args:
+        TARGET: ubuntu:devel
+      dockerfile: debian/Dockerfile

docker-testing/debian/Dockerfile

@@ -0,0 +1,14 @@
+ARG TARGET
+
+FROM ${TARGET}
+
+ARG TARGET
+ENV WEST_TARGET=${TARGET}
+
+RUN apt-get update \
+    && apt-get install -y \
+        git \
+        python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install --break-system-packages tox

docker-testing/fedora/Dockerfile

@@ -0,0 +1,13 @@
+ARG TARGET
+
+FROM ${TARGET}
+
+ARG TARGET
+ENV WEST_TARGET=${TARGET}
+
+RUN dnf install -y \
+        git \
+        python3-pip \
+    && dnf clean dbcache
+
+RUN pip3 install tox

docker-testing/in-container-test.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+set -e
+
+# This is the test script that runs in the containers themselves.
+
+WEST=/west
+# Replace semicolon with dash
+WEST_TARGET=${WEST_TARGET//:/-}
+
+WEST_TOX_OUT=$WEST_TOX_OUT/$WEST_TARGET
+WEST_TOX_OUT_IN_HOST=$WEST_TOX_OUT_IN_HOST/$WEST_TARGET
+
+die() {
+    if [ $# -eq 0 ]; then
+        echo "error: $*" >&2
+    else
+        echo "error: unknown error in $0" >&2
+    fi
+    exit 1
+}
+
+main()
+{
+    # Verify the container environment set up meets this script's requirements.
+    [ -n "$WEST_TOX_OUT" ] || die "missing $WEST_TOX_OUT"
+    [ -n "$WEST_TOX_OUT_IN_HOST" ] || die "missing $WEST_TOX_OUT_IN_HOST"
+    [ -d "$WEST" ] || die "missing $WEST in the container"
+
+    TOX_LOG="$WEST_TOX_OUT/tox.log"
+    TOX_LOG_IN_HOST="$WEST_TOX_OUT_IN_HOST/tox.log"
+    WEST_TESTDIR="/tmp/west"
+
+    mkdir "$WEST_TOX_OUT"
+
+    git clone -q "$WEST" "$WEST_TESTDIR" || die "failed to clone west to $WEST_TESTDIR in container"
+    cd "$WEST_TESTDIR"
+
+    echo "running tox, output in $TOX_LOG_IN_HOST in host"
+    tox run >"$TOX_LOG" 2>&1 || die "tox failed"
+
+    cp -R htmlcov "$WEST_TOX_OUT" || die "failed to copy coverage to $WEST_TOX_OUT_IN_HOST/htmlcov in host"
+}
+
+main "$@"

docker-testing/run-tests.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# This is the top-level test script that runs in the host.
+
+HERE=$(dirname "$0")
+
+[ -d "$HERE/outdir" ] && rm -r "$HERE/outdir"
+
+set -e
+mkdir "$HERE/outdir"
+export MY_UID=$(id -u)
+export MY_GID=$(id -g)
+export WEST_IN_HOST=$(realpath "$HERE/..")
+# Store the final config as reference
+docker-compose config > $HERE/outdir/config.yml
+docker-compose up --force-recreate --build