Nix daemon blocked by sandbox on macOS Monterey (on external SSD) #6291
Comments
|
I haven't run it on an external disk, but I would imagine someone has. I don't know what this would be off the top of my head, but I guess we can start picking at it. A few Qs:
|
|
Hi, thanks for helping out.
.
|
|
I'm running into the same/similar issue. Been through uninstall procedure (#6144) about 5 times. Booting from external SSD, but not using FileVault. macOS Catalina. |
|
Still just kinda guessing/collecting. I don't recall any ~sandbox reports like this before, but the external disk bit does remind me a little of some issues related to Nix installs on VMs and EC2. We don't fully understand it, but macOS does treat internal and external drives a little differently, so I guess that's the thread to pull on for now.
It might also be worth opening Console.app to see if there are any obviously-correlated messages that give a little more insight. |
|
The darwin-store launch daemon seems to fail each boot. The following is after I run it manually with |
|
I have no experience at all with sandboxing on MacOS, except as an end user. However, something that showed up when I searched was that there seems to be two different privileges, In my System preferences, under Security and Integrity -> Integrity -> Files and Folders, Terminal is listed as having "Removable Volumes", while nix-daemon is listed with "Full Disk Access". (I'm using Swedish so the terminology might be a bit different.) Maybe nix-daemon inherits the Terminal-privileges when I execute it manually? |
|
Since we're exploring I'll try to think out loud for posterity (don't worry too much if something I say doesn't make sense; feel free to ask questions). I'll start with the command outputs (leaving your followup comment on the privileges aside for now):
The difference I note in 2 is out in that territory where we don't exactly understand what the system is doing well enough to be sure if it's a problem or not.
So my thought for the moment is:
|
So no use running Edit: Posted the wrong command execution. Output still the same. |
|
Hmm. I was thinking not, but I just double-checked the manpage for diskutil, and it says enableOwnership will:
That's the same database So--yeah, go ahead and try |
|
No difference after Since I had |
|
Drat. Does |
|
|
Hmm. Can you try the enablePermissions command for /nix as well (and reboot)? I assumed the installer would have done that, but perhaps not. |
|
I just did (including reboot), but still no difference, except for the output of |
|
I have no idea if it could have any impact, but I failed to mention when opening the issue that I had nix installed prior to upgrading to Monterey. So the re-installation failed asking me to remove nix entries in bashrc etc. I did all these things (repeatedly) until the installation went through without errors. While I removed the things I was asked to, I guess that e.g. build users created by the earlier installation might have been reused. (I did remove the new nix volume between each retry.) |
|
Ah. Perhaps. A complete uninstall is tricky and might be confounding. Were you on Catalina? It was working with the same general setup? It might be worth following the not-quite-released uninstall directions from the unstable manual https://nixos.org/manual/nix/unstable/installation/installing-binary.html#macos and giving a reinstall a try. |
|
I was not on Catalina but went straight from Mojave to Monterey. I now performed all the steps from the linked uninstall directions, followed by a reinstall of nix. The issue persists. The volume was mounted after installation but nix-daemon fails to access libsodium during so is not running. Will reboot now. |
|
No luck. No mounted volume, and after manually triggering darwin-store, nix-daemon still fails. |
|
Ok. Makes sense that you leapfrogged Catalina. Back to your permission note earlier, nix doesn't currently use any of the ~Security permissions (and I don't think it can, automatically--IIUC the best anyone can do is trigger an action that would cause their app to show up in the list and then open the panel for the user to toggle it). I would guess that you have added both of these permissions over the years while troubleshooting nix or other software, and the uninstall won't flush them. I think I recall setting full disk for the daemon being a workaround that the same set of EC2/VM users were using to get nix working before the more recent enableOwnership change (which at least appears to have caused the trickle of those reports to dry up). Given that history, it does make sense to me that adding some may be forcing the issue. If you don't already have it installed with the same permission, it might be interesting to install iterm2 and see if manually invoking the daemon from there triggers the sandbox error, or if it just works? Fair warning that I am otherwise running low on actionable hunches. I'll try to do a little more reading myself, but my responses may slow down a little if we hit another dead-end or two. |
|
Thank you so much for all your effort so far! I wouldn't have expected anywhere near this engagement for an edge case like mine, but your response has been better than any paid customer service I have ever used ... I would also like to thank you and your fellow committers for your work on the installer. It has become really great, with clear information on what's going on and helpful instructions when it detects something's off. It so good I almost don't mind having to reinstall nix for the fiftyeleventh time ... I will test with iTerm2, but probably not before the weekend. |
|
Hitting the same issue. Manually running I'm seeing this on osx 12.4, running on an amazon mac2.metal instance (perhaps that's a way that people can repro this). |
As in, running it once permanently fixes it? Or just that it works that time? |
|
It just works that time. So I leave that terminal window open to run the daemon, then in other terminal windows my nix-shell commands will work (I can see the daemon terminal outputting 'accepted connection'). |
|
What do |
|
|
Two thoughts:
|
|
I have a physical mac where nix works fine (the above output is from a mac VM on ec2). Stat'ing the same files, I see on my working physical mac that root.nixbld owns /nix and everything in it. So I think the problem on the mac VM is that the logged-in user is somehow gaining ownership of some files in the /nix/store (or perhaps the inverse like you said: the logged-in user should own everything, not root). Although it's unclear how this relates to the root cause problem of the fs sandbox blocking open() in /nix. Tomorrow, I'll try to manually chown /nix and see if it fixes the issue. |
|
Someone on matrix chimed in and said they do have root/nixbld for both /nix and a copy of nix-daemon. The source indicates that root/nixbld are correct. (Someone on an older macos said they have root/wheel. Not sure what the cause of that one is...) It's possible the installer is holding something wrong. I know it uses sudo a lot and I gather macos sudo is a little weird. If basically everything else in your store is owned by your user (and the same isn't true on your physical Mac), I'd wonder first if aws is using a modified sudo that breaks our assumptions? |
|
I've ran into the same issue on macOS Monterey 12.5: I moved When I granted "Full Disk Access" to I wasn't happy with giving any user access to an executable with full disk access, so I:
After all this, |
|
@YorikSar Am I understanding right that this is on a local system that's easy to poke at? (I don't have fresh ideas atm; just asking for the record since people able/willing to poke at this kind of issue on-demand have been scarce.) |
|
@abathur Nope, it's a server-ish Mac that I spend whole day migrating to external SSD to have more space in Nix store :) So all poking requires a maintenance window and workflow disruption for others. |
|
Drat :) |
|
It repro'd for me every time on an aws ec2 mac.
…On Fri, Dec 23, 2022, 4:00 PM Travis A. Everett ***@***.***> wrote:
Drat :)
—
Reply to this email directly, view it on GitHub
<#6291 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAXDM2CM7XE4LOKRFBL345DWOY4JRANCNFSM5RFHZHFQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I confirmed that on x86_64-darwin macos ventura nix daemon runs without issue even with external SSD drive (this happens to be aws mac1.metal). However on arm/aarch64 darwin mac2.metal instance running ventura, we see the same issue reported here |
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-04-14-nix-team-meeting-minutes-48/27358/1 |
|
@samrose @YorikSar @sheldonneuberger @bryanhonof Previously I mentioned opening Console.app to look for anything that correlates, and this morning I read a blog post that mentions sandbox denials being logged--so it's possible that some of you maybe be able to see a message there (or via the (You should just be able to search "sandbox" to see all such messages, but narrowing the predicate or closing all other apps may make it easier to find the right ones. If you've added an exemption in Security & Privacy for these, you'll need to disable it and maybe reboot?) I'm not quite sure it'll apply quite the same in our case since the blog post is specifically focused on apps, but https://ubrigens.com/posts/sandbox_coverage.html and the associated code in https://github.com/0xbf00/macos-sandbox-coverage may be a thread worth pulling on to see if we can at least identify the specific sandbox profile/rule that's snagging you. If we know that, maybe we'll be able to walk this back to documentation or code that'll enable us to understand what's going on and whether this is intrinsic or something Nix or the installer could be working around. |
|
I've created a macOS VM in UTM with macOS guest. Thankfully, UTM makes it extremely easy. I've created a file with I don't think there's much else to do here... launchd starts processes with permission context tied to the binary that is being started, and I found this article: https://chrispaynter.medium.com/what-to-do-when-your-macos-daemon-gets-blocked-by-tcc-dialogues-d3a1b991151f - it describes an approach where one binary is running both as daemon from root and as agent from user. It allows to request all necessary permissions from user as agent and they will be applied to the daemon as well. I think that instead of using |
|
Seeing the same thing on Ventura 13.2.1 on ec2 (mac2.metal). For any EC2 users out there, you can work around this by signing in over VNC and granting Full Disk Access permission to nix. Here is the step-by-step:
I'm not aware of a way to do this in a fully automated way, sadly. But the good news is that once you do this, you can create a custom AMI and use that for launching all your future instances. |
|
@YorikSar, I assume this aligns with your findings? I.e. |
|
Sigh, this has been a bit annoying, lol... so here it goes. I'm using MDM to provision the Macs, and I'm using an external disk for nix (because it's bigger, cheaper, and reduces the wear on the internal small one). and with (as root) we now have a launcher that resides on the system disk (🤦). Changing and installing a TCC profile (I'm using mosyle), with Application Path: as You can probably set this by hand as well to only that one executable and be done. It's still rather stupid to hack around like this. |
for the lazy 🙄 |
Battled for a weekend, finally seems to solve the problem. |
|
I found a hacky workaround for people on EC2: Install https://tart.run on the instance, spawn a VM, then SSH into the VM and install Nix there. It's using Virtualization Framework so it's fast (native perfomance). This can all be done completely non-interactively. |
Describe the bug
After installing nix the nix daemon won't run since it's being blocked by sandboxing from opening files in the nix store. This is happening on macOS Monterey installed on and booted from an external SSD.
Steps To Reproduce
sh <(curl -L https://nixos.org/nix/install)ps -ef | grep nix. It's not. (Occasionally a wait4path is running if darwin-store daemon hasn't started correctly.)tail /private/var/log/nix-daemon.log(Repeated over and over again.)
So libsodium can't be loaded due to sandbox rules.
Expected behavior
nix-deamon and darwin-store running
nix-env --versionoutputAdditional context
Note that I'm running MacOS on a FileVault encrypted external SSD (with the internal disk locked). I don't know for sure that this is related to the problem, but as it's probably not the most usual setup it might be the reason why I encounter this issue while others can install Nix on MacOS without problem.
The text was updated successfully, but these errors were encountered: