podvm-mkosi: use toolchain from nixpkgs

[katexochen]

If we want to have reproducible OS image build, we need to use the toolchain from nixpkgs. The mkosi version in nixpkgs uses a patched version of systemd (with commits that are upstream, but will only be part of the next release). We also fixed many of the tools mkosi uses to get things reproducible.
I maintain the mkosi package in nixpkgs with a colleague. We have nightly tests running on nixpkgs master + mkosi main to ensure things stay reproducible and working.

The use of Nix in general is discussed in #1516. Nevertheless, as I already have this ready, I wanted to open this at least as a draft. I don't think using Nix must be an all-or-nothing decision. I don't think there is currently a way to get around using it for what it is used in this PR if we want reproducible images.

[mkulke]

LGTM. built an image and tested it in a peerpod deployment.

I didn't test the GH action code. I'm not entirely sure about the build-space issue on a worker, locally my podvm-mkosi folder consumes 2.5G, and this includes the vhd copy. I'd suspected docker build leftovers, but we should be on a fresh node 🤔

[bpradipt]

I tried the gh workflow using nektos/act and it failed during the build image step.
Binaries got built successfully.

warning: Git tree '/home/ubuntu/cloud-api-adaptor' is dirty
error: getting status of '/nix/store/7pqci4rwn43ng88610r6a79xxx1j2irm-source/flake.nix': No such file or directory
[Create a Pod VM image with mkosi/Build image   ]   ❌  Failure - Main Build nix shell to cache dependencies
[Create a Pod VM image with mkosi/Build image   ] exitcode '1': failure
[Create a Pod VM image with mkosi/Build image   ] ⭐ Run Post Install Nix
[Create a Pod VM image with mkosi/Build image   ]   🐳  docker cp src=/home/ubuntu/.cache/act/cachix-install-nix-action@v22/ dst=/var/run/act/actions/cachix-install-nix-action@v22/
[Create a Pod VM image with mkosi/Build image   ]   ✅  Success - Post Install Nix
[Create a Pod VM image with mkosi/Build image   ] 🏁  Job failed
Error: Job 'Build image' failed

[bpradipt]

Another question on the gh-action, is it possible to reuse a pre-built binaries for the podvm "build image" job ? In my local env, re-running the "build image" job results in rebuilding the binaries. And it took roughly 30 min to build the binaries on a VM with 2vcpus and 16GB mem.

[katexochen]

I tried the gh workflow using nektos/act and it failed during the build image step. Binaries got built successfully.

I haven't tried this with act, not sure what caused the error you observed. If you want to test this workflow, you can just trigger it on your fork. Container images are automatically pushed to the registry of the repo the workflow is executed. Here is a successful run on my fork: https://github.com/katexochen/cloud-api-adaptor/actions/runs/6928972339

[katexochen]

Another question on the gh-action, is it possible to reuse a pre-built binaries for the podvm "build image" job ? In my local env, re-running the "build image" job results in rebuilding the binaries. And it took roughly 30 min to build the binaries on a VM with 2vcpus and 16GB mem.

Yes, this definitely can be optimized. Should we just add a input for the image name? Then we can skip the first job if the image is passed as input.

As we've already discussed a few times, we should get rid of the binaries container image build step, build binaries in separate and cache them in the mid-term.

[bpradipt]

Another question on the gh-action, is it possible to reuse a pre-built binaries for the podvm "build image" job ? In my local env, re-running the "build image" job results in rebuilding the binaries. And it took roughly 30 min to build the binaries on a VM with 2vcpus and 16GB mem.

Yes, this definitely can be optimized. Should we just add a input for the image name? Then we can skip the first job if the image is passed as input.

Yeah, this might be helpful.

As we've already discussed a few times, we should get rid of the binaries container image build step, build binaries in separate and cache them in the mid-term.

Would it make sense to split the workflow into two - one for building the binaries and another for the podvm image ?

[katexochen]

Would it make sense to split the workflow into two - one for building the binaries and another for the podvm image ?

Not sure about this, that would place the burden on managing cache invalidation on the developer - which hasn't been working out well with the old image pipeline, where we would only update these images once or twice a release cycle.

[bpradipt]

Would it make sense to split the workflow into two - one for building the binaries and another for the podvm image ?

Not sure about this, that would place the burden on managing cache invalidation on the developer - which hasn't been working out well with the old image pipeline, where we would only update these images once or twice a release cycle.

I see..

[katexochen]

Workflow run with prebuilt binaries image: https://github.com/katexochen/cloud-api-adaptor/actions/runs/7043741095/job/19170078950

[bpradipt]

nice.. thanks for adding this.

[bpradipt]

/lgtm
Thanks @katexochen