Content-addressed realisations can't always be registered

[Mindavi]

Describe the bug

I keep running into issues with building content-addressed derivations using hydra. This may be related to hydra, but I can reproduce this issue with just nix-build as well.

failure building ‘/nix/store/bl38jaxgf2afjpi9ylrdk3rywv0b2dfm-glibc-aarch64-unknown-linux-gnu-2.33-49.drv’ on ‘localhost’: error: Trying to register a realisation of 'sha256:5d2cb0ce72b94b4331aea2f2c17b297472e1481e877785ddb7fd243d94ae4fb9!bin', but we already have another one locally.

  Local:  /nix/store/7506l8lijp27ppg2h7d3kp4clf20r3gn-glibc-aarch64-unknown-linux-gnu-2.33-49-bin
  Remote: /nix/store/qfhkagq1qvlfsmvg946z59qph8av65b5-glibc-aarch64-unknown-linux-gnu-2.33-49-bin

This is the diff diffoscope gives between the two glibcs: glibc-diff.html.txt

Differences I see:

• different build-id
• different dynamic linker path (comes from itself, I guess)

It's unclear to me why a build is rejected and how I can prevent these errors from happening. A gc helps but is a big hammer to work around this issue, since it can lead to having to do a lot of rebuilds.

Steps To Reproduce

Running a hydra instance against staging. I guess these different glibc derivations are caused by changes in the stdenv? I'm not sure.

I can't really give a clear path to full reproduction, but I can debug this further whenever it (inevitably) happens.

Expected behavior

Builds just work without having mismatches. If the derivation gives the same (ultimate) result, it could 'just' be substituted after finding out that it's the same result.

nix-env --version output

nix-env (Nix) 2.4pre20210802_47e96bb

Additional context

Using a fork from hydra by @regnat, the PR is WIP, so it may be related to that. Since I get the build error when building glibc by hand (after this happens in hydra) as well, it's unclear if this is a nix, nixpkgs or hydra issue.

[Ericson2314]

Could it be there is an upstream non-deterministic dependency. What derivation(s) do those dynamic linker paths correspond to?

[Mindavi]

Oops, attached the wrong diffoscope output (from a previous time this issue occured). This should be the correct one:
glibc-diff2.html.txt

glibc1 (local): /nix/store/lmn4wd7ixm7fs0kp8afxnfypnyvl0l6v-glibc-aarch64-unknown-linux-gnu-2.33-49/lib/ld-linux-aarch64.so.1

glibc2 (remote): /nix/store/gzzcgxgzqk6zj72p35c2dk9l42axpw1x-glibc-aarch64-unknown-linux-gnu-2.33-49/lib/ld-linux-aarch64.so.1

The build ID also differs between the two.

Otherwise they seem to be identical, as far as I can tell. It may be that that build ID is generated in a way that can differ between builds?

Edit: I'm not sure if this is the answer you wanted, maybe you could give me a hint on how to find the derivation they correspond to?

[thufschmitt]

It looks like the hydra branch does something funky when the derivation is already realised locally (it shouldn’t try to fetch it from the cache at all, but it still apparently does). At the same time, the build of glibc seems is non-determistic (https://r13y.com seems to agree on that too). The combination of both causes hydra to try to register another glibc than the existing one.

But overall, the real bug is on the hydra side (it should just re-use the glibc that it knows locally and don’t even try to reach the cache). So I’ll defer this to NixOS/hydra#875 (sorry for not answering there btw, I was convinced that I did last week but apparently forgot)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881/206