hercules-ci-nix-daemon processes leftover after effect task completes

[max-privatevoid]

Description
On a non-NixOS machine, with Nix installed in single user mode and Hercules CI Agent installed in .nix-profile, executed via a systemd user service, there seem to be hercules-ci-nix-daemon processes sticking around indefinitely after effect tasks are completed. This problem may also exist on NixOS, but may be less noticeable thanks to hercules-ci-agent-restarter.

The effects in question are two runNixOS effects from https://github.com/privatevoid-net/privatevoid-infrastructure

To Reproduce
Can't reliably reproduce.

Expected behavior
No leftover hercules-ci-nix-daemon processes.

Logs
There seem to be no relevant logs, but this is what the processes look like.

● hercules-ci-agent.service
     Loaded: loaded (/var/lib/nixbuilder/.config/systemd/user/hercules-ci-agent.service; static)
     Active: active (running) since Tue 2022-09-13 21:57:33 CEST; 1 weeks 0 days ago
    Process: 3470412 ExecStartPre=/var/lib/nixbuilder/.nix-profile/bin/hercules-ci-agent --config /var/lib/nixbuilder/.config/hercules-ci/agent.toml --test-configuration (code=exited, status=0/SUCCESS)
   Main PID: 3470435 (.hercules-ci-ag)
     CGroup: /user.slice/user-984.slice/[email protected]/hercules-ci-agent.service
             ├─1818684 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-837.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─1818685 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-836.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2575176 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-885.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2575250 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-886.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2641075 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-905.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2641089 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-906.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2644819 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-920.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─2644846 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-919.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─3470435 /var/lib/nixbuilder/.nix-profile/bin/hercules-ci-agent --config /var/lib/nixbuilder/.config/hercules-ci/agent.toml
             ├─3580436 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-19.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─3580521 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-20.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             ├─3744459 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-107.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1
             └─3744547 /nix/store/jpjcbqcad6m0f833j0crsixlj9v4jfrw-hercules-ci-agent-0.9.7/bin/hercules-ci-nix-daemon --option netrc-file /var/lib/nixbuilder/.cache/hercules-ci/work/secure/tmp-netrc3470435-108.key --option substituters s3://nix-store?endpoint=object-storage.privatevoid.net https://cache.nixos.org/ https://cache.privatevoid.net/ https://max.cachix.org?priority=60 --option trusted-public-keys cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.privatevoid.net:SErQ8bvNWANeAvtsOESUwVYr2VJynfuc9JRwlzTTkVg= max.cachix.org-1:oSMQ1zYLR8H4L17hfe6ETlI/d+VeiBykB8PbBdPtDJw= --option narinfo-cache-negative-ttl 0 --option max-jobs 1

Platform / Version
0.9.7

[roberth]

Found two problems

• Does not work with async exceptions haskell/process#107 and
• we'll want more than SIGTERM https://github.com/haskell/process/blob/25613c7d01f6fe5ee2f75fd39db898942cbfb404/cbits/posix/runProcess.c#L202