CI builds for darwin

[cleeyv]

Last week I finished initial implementation of CI evaluations and builds of ngipkgs on each push and PR using github actions: #18. However, it was implemented to only build on linux (all the monorepo dev work thus far had been done on NixOS). After sharing an update on my progress, I received the following feedback from @fricklerhandwerk:

Albert's mob asked about darwin builds in the CI
Is there a particular reason you only have x86_64-linux?
Apparently most of them are on darwin

I replied at the time with my partial understanding at the time of the problems with darwin builds of ngipkgs. Since then I have worked on it from various angles and have a much clearer understanding of the problems involved and potential solutions.

Background Info

One foundational piece of nix knowledge that is at the core of understanding the problem is the difference between evaluation (which generates a derivation) and realisation (aka building), which is the carrying out of the build instructions in the derivation to produce an output in the nix store.

When using flakes, every derivation (package, test, dev shell, etc) needs to have a system attribute specified (defining the CPU architecture and the OS, such as x86_64-linux or aarch64-darwin) and all the dependencies of the derivation need to be available for the matching system. A derivation will have an evaluation-time error if one of its dependencies isn't available for its system.

Since there are so many potential systems that nix can run on, for multi-platform software manually defining system attributes in a flake would require a lot of boilerplate code. Hence the widespread use, including in the ngipkgs flake, of flake-utils which allows for assigning system attributes for a list of defaultSystems, which includes the following: ["x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin"].

CI of a nix flake can do two different kinds of tests: evaluating the derivations of the flake, and building them. As I explained in #18, evaluating all the derivations is well-supported with the nix flake check command, but there is no supported way to build all of the derivations. However, nix flake check does build all of the derivations located in the checks attribute, so adding all packages into checks results in them being built, making this a suitable command for a CI test of both evaluation and building. Importantly, the evaluation phase of nix flake check uses all of the potential systems (defaultSystems) of the flake, whereas the build phase in checks only uses the single system that the build is currently running on.

CI Problems

There were a few different problems standing in the way of getting nixpkgs CI working for darwin systems. Some of them have been resolved, and at least one of them has not.

First, one of the initial packages in ngipkgs (liberaforms) is built using mach-nix, which requires doing a build of an intermediary derivation as part of the evaluation process of creating a final derivation for the package. This kind of "build at evaluation-time" process is known as an "Import From Derivation" (IFD).

The problem is that on a flake designed to be used on more than one system, there is an incompatibility between IFD and nix flake check. The check command attempts to evaluate all possibilities defined in the flake, including all of the defaultSystems from flake-utils, and then when it hits the IFD it tries to do the intermediary build for all those systems as well. Of course, it immediately hits an error when trying to run a build for a system that is not supported by the current hardware (there is an an issue for this: NixOS/nix#4265). I had initially worked around this problem and gotten the CI working, despite the liberaforms IFD, by configuring flake-utils to only support x86_64-linux systems, which is what led to the initial feedback from @fricklerhandwerk and @albertchae.

Second, to continue testing the viability of running a nix flake check on darwin CI, I removed liberaforms from ngipkgs so there would be no more IFD-induced evaluation-time system build errors. However, there was now a new evaluation-time system error! It informed me that for one of the remaining initial packages on ngipkgs, libgnunetchat, there was a dependency from nixpkgs, gnunet, that is not available on one of the darwin host platforms requested by the defaultSystems list. Specifically, its default.nix expression contains the following attribute: platforms = platforms.gnu ++ platforms.linux; which means it can run on many CPU architectures, but only on systems with a linux kernel (not darwin).

Third, since this was another evaluation-time error for compatibility with many defaultSystems, and the main thing I wanted to do on CI was build on a single darwin system, I thought I would try the workaround suggested in the nix error message: bypass this evaluation-time check of system compatibility using the environment variable NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 and run nix flake check with the --impure option to give it access to the environment variable. This allowed the evaluation to complete. And when I ran it locally the build completed as well (on a x86_64-linux system). Finally, I ran it on actual darwin hardware via a macos-latest CI job I had configured on github actions, and it failed on a build error which in retrospect was not surprising since it was trying to do a build on macos that included a dependency that is linux-only. In short, the evaluation error had been correct all along and bypassing it to run the build anyways just confirmed this.

Potential Solutions

There aren't currently any packages in the repo that are darwin-compatible. Once one arrives, then there will be a more practical basis for testing potential solutions. Based on the latest brain dump last night from the moss mob in #10 it sounds like the flarum package they're working on in moss-flarum-2 branch is recently building successfully. Since many of them are on darwin, it seems plausible that I could use that branch for testing darwin CI. I see that branch currently has a CI error I haven't seen before, so I'll have to look into that first.

To simplify the problem, if I put aside the evaluation phase for now (which can easily be done in practice using NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM) then the core solution for this issue seems like it could be adding some system-based conditional logic to the checks phase of the flake so that linux-only packages don't build when running on a darwin system. Or alternatively, that only darwin-compatible packages build when running on a darwin system.

There is also the related but secondary problem of liberaforms and future potential IFDs. As discussed in #10, in addition to the nix flake check CI incompatibility discussed above, each package in the repo with an IFD also adds a performance penalty for the evaluation time of the entire flake. For both of these reasons, as I said in the other issue, "I understand why nixpkgs doesn't allow them and I think it could be prudent to consider this rule for ngipkgs as well". For now, while the flake remains linux-only, I'll leave liberaforms enabled. However if I'm successful in resolving the above problems to get darwin CI working then I would remove liberaforms as part of merging that into the main branch.

[albertchae]

@cleeyv thanks for your investigation into darwin builds. One thing I want to clarify is that we (@ngi-nix/moss) have actually been avoiding building anything on darwin since our first meeting, initially by running x86_64-linux VMs with lima and now through a shared x86_64-linux build server set up by @jleightcap. So we haven't explicitly been testing whether flarum is compatible with darwin (but also there's no reason it shouldn't be compatible? we just haven't tried).

[cleeyv]

@albertchae thanks for the update! Perhaps I misunderstood the need for darwin CI motivating this issue: based on your work so far with moss, do you think it would be useful for there to be CI builds of PRs for darwin?

I'm curious what the motivation was in the first place for doing all dev work on x86_64-linux - whether this seemed like an unfortunate workaround done out of necessity or something that you think makes sense to continue doing (and that you would expect other teams would do as well). Based on how you're operating now, it sounds like darwin CI wouldn't be immediately relevant, but do you expect or want that to change in the future?

These are bigger conversations for Summer of Nix and ngipkgs that I'm going to raise with others as well. I've never used nix on darwin, and it never came up in my past experiences as a Summer of Nix participant, so I don't have so much of my own experience to draw on for this question.

So we haven't explicitly been testing whether flarum is compatible with darwin (but also there's no reason it shouldn't be compatible? we just haven't tried).

Based on a quick look through the flarum documentation, I get the impression that in theory it could be run on MacOS but since that server hardware is relatively rare/expensive, practically it is much more likely to be run in production on linux. However, since MacOS is way more common for dev machines, they do mention in their section on Development Tools the use of tools like Laravel Valet to do local dev on Flarum. Based on this, I can imagine the usefulness of a dev shell / local deployment config of some kind for darwin. Though it also sounds like you've already been supporting that use case on x86_64-linux with lima which maybe is simpler. I'm curious to hear if you have further thoughts on this (I'm going to cross-post this over to the flarum issue as well).

[cleeyv]

While looking into integrating nixos tests in ngipkgs I came across this relevant info about work on running them on darwin (still not completed):
https://www.tweag.io/blog/2023-02-09-nixos-vm-on-macos/
NixOS/nixpkgs#193336

[cleeyv]

I read more carefully through the issue about the IFD problem with nix flake check NixOS/nix#4265 and realized that there was a fix for this problem recently released with nix version 2.16! Apparently it doesn't always fix the problem (the problem persists some common Haskell IFDs, for example) but it potentially does. Testing this will require running nix flake check with version 2.16 of nix. It appears that both my local NixOS and nix-installer on the CI are using versions older than 2.16 so the fix has not been tested yet for liberaforms on ngipkgs.

I have upgraded my local system to nix 2.16 and when I ran nix flake check with IFDs enabled, and flake-utils configured with all of the defaultSystems, it works and provides a warning that was introduced in with the fix!

$ nix flake check --allow-import-from-derivation
warning: Git tree '/home/cleeyv/dev/ngipkgs' is dirty
warning: The check omitted these incompatible systems: aarch64-darwin, aarch64-linux, x86_64-darwin
Use '--all-systems' to check all.

This is a bit tangential to CI builds for darwin, but does partially resolve one of the problems encountered along the way and gives more flexibility regarding the question of whether to allow IFDs in ngipkgs or not.

The current default nix version for nix-installer being used on CI is 2.15. However, it should be possible to override the nix package version with an environment variable added to the ci.yaml: DeterminateSystems/nix-installer#387 (comment)

[mightyiam]

For what it's worth, I don't like the limitations imposed by IFD neither during development under this repo or maintenance of it.

[lorenzleutgeb]

Current state of affairs is that we do not support darwin. We're not actively trying to get things packaged for darwin and we do not have CI for it. The additional maintenance effort is currently not worth it.