Nix Flakes is not usable without access to github.com

[bjornfor]

Describe the bug

I want to try/use Nix Flakes on an offline network. I have configured a local registry entry for offline use, but Nix still insists on getting the global registry from github.com, which prevents me from using it (the Flakes feature).

Example:

$ nix --extra-experimental-features "nix-command flakes" run mylocalflake#app
warning: error: unable to download 'https://channels.nixos.org/flake-registry.json': Couldn't resolve host name (6): retrying in 253 ms
[...]
error: unable to download 'https://channels.nixos.org/flake-registry.json': Couldn't resolve host name (6)

Non-Flakes Nix parts work fine on offline network.

Expected behavior

Nix Flakes can be configured with local registry and be usable without access to github.com.

nix-env --version output

nix (Nix) 2.13.5

Priorities

Add 👍 to issues you find important.

[a-h]

I have a reproduction of this error at https://github.com/a-h/nix-issue-8953 - it uses Multipass to create a VM.

First, I define a cloud-init.yaml for the VM that installs Nix, and sets up the default nix registries while online with nix registry list.

It then clones a flake's repo for offline use with nix flake clone, and creates a user redirection of the Github address to the local path it just cloned to with nix registry add.

Finally, it sets up a firewall rule to deny outbound traffic.

#cloud-config
users:
  - default
  - name: nix-issue-8953
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

runcmd:
  - # Install Nix.
  - curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --no-confirm
  - . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
  - # Initialize flakes.
  - nix registry list
  - # Download for offline.
  - nix flake clone "github:numtide/system-manager" --dest /flakes/github.com/numtide/system-manager
  - nix registry add "github:numtide/system-manager" /flakes/github.com/numtide/system-manager
  - # Disable outbound traffic, e.g. to github.
  - iptables -t filter -I OUTPUT 1 -m state --state NEW -j DROP

When I attempt to run the flake, I expect it to not require downloading the system-manager flake's source code, because I've cloned it for offline use, and placed a pointer in the registry.

multipass exec nix-issue-8953 -- sudo bash --login -c "nix run github:numtide/system-manager"

However it does, which is a bug.

multipass exec nix-issue-8953 -- sudo bash --login -c nix run github:numtide/system-manager
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 256 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 683 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 1133 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 2605 ms

See https://github.com/a-h/nix-issue-8953/ for step-by-step instructions.

Notes

In a more realistic offline scenario, I expect to need to use nix copy to copy flake outputs and derivations from a machine with Internet access (where the Internet access is used to download source code and binary cache outputs) to a machine without Internet access.

I expected nix copy to copy flake source code too, but apparently that has to be done via the nix flake clone and nix registry add commands as an extra step, and it doesn't work as of Nix 2.17.0.

There's a chat where I explore this issue at https://matrix.to/#/!KqkRjyTEzAGRiZFBYT:nixos.org/$52VhywMgI8f2h-kOTjl6shkksIC997sqSXnyioey0-U?via=fairydust.space&via=matrix.org&via=tchncs.de

[Artturin]

I have a reproduction of this error at a-h/nix-issue-8953 - it uses Multipass to create a VM.

First, I define a cloud-init.yaml for the VM that installs Nix, and sets up the default nix registries while online with nix registry list.

It then clones a flake's repo for offline use with nix flake clone, and creates a user redirection of the Github address to the local path it just cloned to with nix registry add.

Finally, it sets up a firewall rule to deny outbound traffic.

#cloud-config
users:
  - default
  - name: nix-issue-8953
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

runcmd:
  - # Install Nix.
  - curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --no-confirm
  - . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
  - # Initialize flakes.
  - nix registry list
  - # Download for offline.
  - nix flake clone "github:numtide/system-manager" --dest /flakes/github.com/numtide/system-manager
  - nix registry add "github:numtide/system-manager" /flakes/github.com/numtide/system-manager
  - # Disable outbound traffic, e.g. to github.
  - iptables -t filter -I OUTPUT 1 -m state --state NEW -j DROP

When I attempt to run the flake, I expect it to not require downloading the system-manager flake's source code, because I've cloned it for offline use, and placed a pointer in the registry.

multipass exec nix-issue-8953 -- sudo bash --login -c "nix run github:numtide/system-manager"

However it does, which is a bug.

multipass exec nix-issue-8953 -- sudo bash --login -c nix run github:numtide/system-manager
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 256 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 683 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 1133 ms
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 2605 ms

See a-h/nix-issue-8953 for step-by-step instructions.

Notes

In a more realistic offline scenario, I expect to need to use nix copy to copy flake outputs and derivations from a machine with Internet access (where the Internet access is used to download source code and binary cache outputs) to a machine without Internet access.

I expected nix copy to copy flake source code too, but apparently that has to be done via the nix flake clone and nix registry add commands as an extra step, and it doesn't work as of Nix 2.17.0.

There's a chat where I explore this issue at matrix.to/#/!KqkRjyTEzAGRiZFBYT:nixos.org/$52VhywMgI8f2h-kOTjl6shkksIC997sqSXnyioey0-U?via=fairydust.space&via=matrix.org&via=tchncs.de

Please post the output of the nix run with -vvvvvvv.

[a-h]

+ multipass exec nix-issue-8953 -- sudo bash --login -c nix run github:numtide/system-manager -vvvvvvv
did not find cache entry for '{"name":"source","type":"file","url":"https://api.github.com/repos/numtide/system-manager/commits/HEAD"}'
downloading 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'...
download thread waiting for 10000 ms
starting download of https://api.github.com/repos/numtide/system-manager/commits/HEAD
curl: Couldn't find host api.github.com in the /etc/nix/netrc file; using defaults
download thread waiting for 100 ms
curl: Could not resolve host: api.github.com
curl: Closing connection 0
finished download of 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'; curl status = 6, HTTP status = 0, body = 0 bytes
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 298 ms
download thread waiting for 10000 ms
download thread waiting for 297 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
starting download of https://api.github.com/repos/numtide/system-manager/commits/HEAD
curl: Couldn't find host api.github.com in the /etc/nix/netrc file; using defaults
download thread waiting for 100 ms
curl: Could not resolve host: api.github.com
curl: Closing connection 1
finished download of 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'; curl status = 6, HTTP status = 0, body = 0 bytes
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 544 ms
download thread waiting for 10000 ms
download thread waiting for 543 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
download thread waiting for 0 ms
starting download of https://api.github.com/repos/numtide/system-manager/commits/HEAD
curl: Couldn't find host api.github.com in the /etc/nix/netrc file; using defaults
download thread waiting for 100 ms
curl: Could not resolve host: api.github.com
curl: Closing connection 2
finished download of 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'; curl status = 6, HTTP status = 0, body = 0 bytes
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 1205 ms
download thread waiting for 10000 ms
download thread waiting for 1204 ms
starting download of https://api.github.com/repos/numtide/system-manager/commits/HEAD
curl: Couldn't find host api.github.com in the /etc/nix/netrc file; using defaults
download thread waiting for 100 ms
curl: Could not resolve host: api.github.com
curl: Closing connection 3
finished download of 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'; curl status = 6, HTTP status = 0, body = 0 bytes
warning: error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6); retrying in 2005 ms
download thread waiting for 10000 ms
download thread waiting for 2004 ms
starting download of https://api.github.com/repos/numtide/system-manager/commits/HEAD
curl: Couldn't find host api.github.com in the /etc/nix/netrc file; using defaults
download thread waiting for 100 ms
curl: Could not resolve host: api.github.com
curl: Closing connection 4
finished download of 'https://api.github.com/repos/numtide/system-manager/commits/HEAD'; curl status = 6, HTTP status = 0, body = 0 bytes
download thread waiting for 10000 ms
error:
       … while fetching the input 'github:numtide/system-manager'

       error: unable to download 'https://api.github.com/repos/numtide/system-manager/commits/HEAD': Couldn't resolve host name (6)
download thread shutting down
xc: exit status 1

[tejing1]

You can configure a different location to fetch the global registry. Since I don't need the global registry at all (and I'd rather not waste time on a network round trip for almost every nix command), I did this in my nixos config:

{
  nix.settings.flake-registry = toFile "global-registry.json" ''{"flakes":[],"version":2}'';
}

[a-h]

I've tested the above setup with Nix 2.16 and Nix 2.13.2 to see if this was a regression added in Nix 2.17, but it wasn't.

So, this behaviour has been in place for some time.

Workaround to stop downloading flake-registry.json

It is possible to pre-configure Nix with a flake registry which prevents it from attempting to download the flake-registry.json file.

In my cloud-init.yaml I run this:

echo "flake-registry = /etc/nix/flake-registry.json" >> /etc/nix/nix.conf

Then remotely, I can push a pre-downloaded flake-registry.json into the target.

scp $PWD/flake-registry.json worker@$WORKER_IP:/home/worker/registry.json
ssh worker@$WORKER_IP 'sudo mv /home/worker/registry.json /etc/nix/flake-registry.json'
ssh worker@$WORKER_IP 'sudo chown root:root /etc/nix/flake-registry.json'
ssh worker@$WORKER_IP 'sudo chmod 664 /etc/nix/flake-registry.json'
ssh worker@$WORKER_IP 'sudo systemctl restart nix-daemon'

This is an ugly workaround, and maybe a better solution would be to include a flake-registry.json file in the installation.

This doesn't fix the problem

Although this stops the first problem of nix registry list not showing anything at all, it appears that the user and global flake registries are still ignored by nix run and other commands, which seems like a fairly serious problem.

I'd like to see:

• Clarity in the logs about the decisions made - what registry files were looked at, what the query was, and ultimately, why the decision was made to download from the Internet, or look at local storage etc. There's not enough information for a lay person to debug.
• Example - clarity around decision to download NAR or other files. From reading the logs, I have no idea why Nix decided it needed to download nar files. I've ran a nix copy operation to copy both derivations and build outputs to a target machine - it starts trying to download files without saying why. Some log data about why Nix thinks it needs to download something would be useful.
• Separate to this isssue, but it seems like nix copy doesn't actually copy flake inputs like source code to the target. i.e. I can run nix copy --to me@other github:user/test to a remote machine, but when I'm on that machine, I run nix run github:user.test the machine attempts to download "stuff" but there's no logs that explain why, so my attempts to debug this via logs have been unsuccessful. Basically, I expect it to be possible to transfer a flake to another machine and run it.

[tejing1]

Separate to this isssue, but it seems like nix copy doesn't actually copy flake inputs like source code to the target. i.e. I can run nix copy --to me@other github:user/test to a remote machine, but when I'm on that machine, I run nix run github:user.test the machine attempts to download "stuff" but there's no logs that explain why, so my attempts to debug this via logs have been unsuccessful. Basically, I expect it to be possible to transfer a flake to another machine and run it.

See #6895.

[Aleksanaa]

{
  nix.settings.flake-registry = toFile "global-registry.json" ''{"flakes":[],"version":2}'';
}

nix.settings.flake-registry = "" will also work.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-implement-dependency-retrieval-primitive/43418/5