Sketch for building Debian source packages in datalad-tracked containers

[mih]

This is part of a larger effort to run an operation like neurodebian using datalad datasets to capture and deploy everything involved.

Here is the part on building packages in containerized environments.
(uses the annexremote package as a mere example)

# TODO:
# - pre-setup gitignore and gitattributes

# this brings "code" under git
datalad create -c yoda builder
cd builder
mkdir envs

# put following in a def file (code/debian-bullseye.def)
###

Bootstrap:docker
From:debian:bullseye

%post
    apt-get -y update
    apt-get -y install --no-install-recommends build-essential devscripts eatmydata equivs

    # remove everything but the lock file from
    # /var/cache/apt/archives/ and /var/cache/apt/archives/partial/
    apt-get clean

    mkdir /build

###
datalad save -m "Bullseye definition"

# track how the container image was built
datalad run -i code/debian-bullseye.def -o envs/debian-bullseye.sif -m "Create SIF" sh -c 'sudo singularity build {outputs} {inputs}'

# put following in code/build.sh
###
#!/bin/sh

set -e -u -x

# somehow -C causes 755 by default
chmod 777 /tmp

eatmydata apt update -y
eatmydata apt upgrade -y

# all building will take place inside the container
# maybe add option to do it on the host FS?
mkdir -p /tmp/build

dcmd cp $1 /tmp/build
(cd /tmp/build && dpkg-source -x *.dsc source)

(cd /tmp && mk-build-deps -t 'eatmydata apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y' -i -r /tmp/build/source/debian/control)

(cd /tmp/build/source && debuild -uc -us -b)

dcmd cp /tmp/build/*changes /build/built
###

chmod +x code/build.sh
datalad save -m "Add build script" code


# setup the untracked locations in the worktree
# TODO consider using a persistent overlay with singularity
# TODO this is likely to be redone by a shim script that does the
# singularity call inside
# this will put FS operations into a file and not cause
# root-owned files to be left behind
mkdir -p cache/var/lib/apt cache/var/cache/apt

echo "cache" > .gitignore
datalad save -m "Ignore cache results" --to-git

datalad containers-add -d . -i envs/debian-bullseye.sif --call-fmt 'singularity exec -B cache/var/lib/apt:/var/lib/apt,cache/var/cache/apt:/var/cache/apt,.:/build --pwd /build -f -e -C -w --no-home {img} /build/code/build.sh {cmd}' build-debian-bullseye

Basic usage would be to "import" a source package from elsewhere ( a queue dataset) and run the build pipeline on it.

mkdir build
#dcmd cp sourcepackage from elsewhere
#datalad save -m "Queue source package to build" build
# TODO this would be a datalad-run call that generates a source package from the source repo

mkdir built

# TODO -o (but that may actually complicated and possibly not even that helpful
datalad containers-run -n build-debian-bullseye $(for f in $(dcmd build/annexremote_1.6.0-1.dsc); do printf '-i %s ' "$f"; done) build/annexremote_1.6.0-1.dsc

[mih]

This is now fully superseded by #12, which has a more complete and neater implementation