[aws mac1.metal][nix] builkdite-agent behaves differently when run locally vs run via launchctl

[OliverKoo]

Hi I am using buildkite on my aws mac1.metal instances. These agents acts as CI agents in our org. Our build process requires nix. I observe the following behavior.

Instances now launch buildkite-agent when the instance is booted (done vis plist in /Library/LauchDaemon). Here is my plist.

When agent execute a job with simple build step command "nix --version" it fails.

But running the same job with agent that I launch locally. (ssh into the machine, then run . [/Users/buildkite-agent/.nix-profile/etc/profile.d/nix.sh](https://github.com/NixOS/nix/blob/master/scripts/nix-profile.sh.in) sets nix related env (also set in plist), then run buildkite-agent start. In this case "nix --version" works just fine.

Seems like launchd jobs have different permission compare to agent launch via normal gui login session. Nix starting from Catalina rely's on mounting a APFS volume at root since root is no longer writable. that vol is own by builldkite-agent user tho. (@abathur is the nix wizard I been chatting with, I will cc him here since knows a lot more about nix than I do)

[ticky]

Hi @OliverKoo, we've not used nix ourselves, so this is pretty new for us!

Just to confirm, has the buildkite-agent binary been given full-disk access in macOS' security & privacy settings?
Is buildkite-agent definitely running as the same user in your two tests?

[OliverKoo]

Hi @ticky thanks for getting back to me

I am not sure if the bk binary has full-disk access. Seems like it has to be granted via GUI from System Prefrence? These are CI machines so there isn't a GUI. Do you know off top of your head how to check or grant access for bk binary?

[ticky]

That's the only way I know of, I don't think there's a workaround for that as it's an OS-level security function.

[OliverKoo]

after giving the agent FDA it worked. thank you! I will close this task