Can't seem to perform remote builds with Darwin

[nixy]

I have looked through quite a few issues on this already, but it seems like the remote builds on Darwin don't work as expected. I would expect that once NixOps sets up SSH access it could use a remote box using the "none" provider, that is to say already running NixOS, as a build machine. But instead NixOps attempts to perform the build locally on Darwin hitting assertions that fail because well it's not Linux.

Here are some of the other issues that seem relevant:

• Support building fully remotely #260
• Create containers remotely when currentSystem != Linux #412
• nixops does not provide nix-build with SSH credentials for OS X-based builds #483

It seems that a lot of these seem to refer to a "workaround" that I can't seem to find any documentation for. I though that using $NIX_REMOTE_SYSTEMS I would be able to defer the builds to the remote systems that are already established, but this doesn't seem to work. I imagine I am doing something wrong with attempting to use distributed builds but there isn't a lot of output to help here:

building all machine configurations...
using predefined remote systems file: [email protected] x86_64-linux /Users/user/.ssh/id_rsa 1 1
error: assertion failed at /nix/store/k7ggnnjn7m9621y9my0lfm2vgbbg3ij3-nixpkgs-17.09pre108299.ec9a23332f/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:32:1
(use ‘--show-trace’ to show detailed location information)
error: unable to build all machine configurations

The line about predefined remote systems file seems to indicate that it is attempting to perform a distributed build but it seems to fail silently and resort to attempting a local build.

Any pointers/advice here? Is using NixOps on Darwin not supported? It seems like this is a showstopper for trying to use NixOps on Darwin if this doesn't work out of the box.

[domenkozar]

The gist is, you need to setup remote builders, so that Nix would realize all store paths on darwin. It's something it could support, but noone has ever written proper support for it, although people did have success setting it up manually.

[nixy]

So at this point I seem to have remote builders set up and have performed a few remote builds. But the remote builds only seem to trigger when I explicitly pass in that the build I am performing is for another architecture. This doesn't seem to happen when I use nixops deploy, so I am wondering if there is a way to force nixops in a similar way?

+$ nix-build -A inetutils
error: Package ‘inetutils-1.9.4’ in ‘/Users/andrew/Development/Nix/nixpkgs/pkgs/tools/networking/inetutils/default.nix:32’ is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

+$ nix-build -A inetutils --argstr system x86_64-linux
/nix/store/5qsmmanp1jasqksn2wwhc3cvbbvzv2v6-inetutils-1.9.4

I would like to document the process for configuring this if I can get it set up. So far this has been pretty frustrating and I would hate to have anyone else duplicate this effort.

[nha]

I would be very interested to know how to set up remote builders @nixy
Did you use https://github.com/holidaycheck/nix-remote-builder (can't get it to work)?

By reading the issues it seems there are several people hitting the same problems (me included), that are already solved (even if only partially).

[domenkozar]

I'm merging all darwin issues into one, so discussion is not spread around.

See #560