complete failure with latest nixosUnstable daemon

[nrdxp]

Description
The agent doesn't seem to like the latest version of nixUnstable, which was recently updated to 2.4pre20210311_8127094 from 2.4pre20201201_5a6ddb3. There has been a minor version bump in the nix protocol from 1.27 to 1.28. I'm assuming whatever changed in the protocol is responsible as the error message indicates that hercules no longer understands the communication from the nix-daemon.

To Reproduce

Deploy a system with the old nixUnstable set as nix.package, a good nixpkgs revision to test is df8e3bd110921621d175fad88c9e67909b7cb3d3. Hercules-ci should work, though it may need a patch for logging.

Now upgrade your system to a revision using the new version. The latest nixos-unstable has the required nixUnstable version. Now hercule-ci-agent will still run, but any derivation it tries to build will fail with the below errors.

Expected behavior

While I would hope it would continue to work as unstable matures, I do acknowledge that we are talking about an unstable nix version here, so I guess I can't exactly demand that it work.

Logs

● hercules-ci-agent.service
     Loaded: loaded (/nix/store/jpy4nbb5y7f2sp0b0q3cdvi3snpfimj3-unit-hercules-ci-agent.service/hercules-ci-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-03-15 18:51:23 MDT; 8min ago
   Main PID: 30311 (.hercules-ci-ag)
         IP: 814.4K in, 6.4M out
         IO: 0B read, 2.7M written
      Tasks: 39 (limit: 4915)
     Memory: 327.5M
        CPU: 16.189s
     CGroup: /system.slice/hercules-ci-agent.service
             └─30311 /nix/store/6yn0p262b2rg338lp9nmwx38jsby51zw-hercules-ci-agent-0.8.0/bin/hercules-ci-agent --config /nix/store/lpv73xxw4d92z4spq3zir6y7c9vnnls4-hercules-ci-agent.toml

Mar 15 18:56:50 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:50][][Info][serval-ws][PID 30311][ThreadId 302][task:e2f9496e-ff31-45f6-9ca3-d6864ecde073][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:189:7] Starting task
Mar 15 18:56:50 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:50][][Error][serval-ws][PID 30311][ThreadId 168][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:7cabd448-dfcc-4edb-b303-331a5b2531db][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 229][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:39a69b47-578b-483e-9a18-6af345fff0e2][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 236][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:5f9c588b-352e-413f-b74d-7e3246f85817][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 243][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:a2ee6f2d-89f4-478a-99f2-4b3aeeb7a599][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 242][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:925508c5-87c3-4482-b3d9-809620920365][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 274][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:b6037e34-b93e-4cb8-80a9-88f7bd63061b][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 302][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:e2f9496e-ff31-45f6-9ca3-d6864ecde073][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:51 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:51][][Error][serval-ws][PID 30311][ThreadId 287][exception:WorkerException {originalException = FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"}, exitStatus = Just (ExitFailure 1)}][task:d7d27f73-c256-4193-bd2e-09c57e7f6dbb][message:FatalError {fatalErrorMessage = "Exception: got unknown message type 1 from Nix daemon; type: nix::Error"} (worker: ExitFailure 1)][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:178:11] Exception in task
Mar 15 18:56:55 serval-ws hercules-ci-agent[30311]: [2021-03-16 00:56:55][][Info][serval-ws][PID 30311][ThreadId 50][task:c40f2429-1bc3-4eef-a15f-c9d08c5d2b61][agent-version:0.8.0][main:Hercules.Agent hercules-ci-agent/Hercules/Agent.hs:171:9] Completed task successfully

Platform / Version

NixOS-unstable/hercules-0.8.0

[roberth]

Well, I guess we can add this to the list of nixUnstable's problems. I've tried to fix issues in nixUnstable end of last year and made some progress, but ultimately it was too broken to be fixed within budget. I'm not too surprised that a new bug appears because upstream is more interested in features than bugs. I can't blame them though, because that's what they get paid for, iiuc.

Anyway, if you want to use the latest Nix for its flakes support, all you really need is the new client, so you can just run the stable daemon. That setup has worked well for me so far. I presume at least some develop on systems with a stable daemon.

That all said, I'll make a start with a branch that builds with nixUnstable, so we can see how bad the other nixUnstable bugs are.

[roberth]

Instructions to try hercules-ci-agent + nixUnstable: #303 (comment)

flake-compat and flake-compat-ci are also an alternative, for those who want to run stable Nix only.

I will introduce a flake-native replacement for ci.nix prototype in the coming weeks.

[roberth]

Upcoming agent 0.8.4 will use Nix 2.4 (and offer a 2.3-based agent package as a contingency). 0.9 will remove the need for a ci.nix. It has come a long way in terms of stability since March, and I'm glad to have contributed some of the memory related fixes.