nixos/users-groups: move home dir creation to systemd tmpfiles

[jsoo1]

Fixes #6481

When the home directory is on a separate mount the user home directories were not created.

Using systemd tmpfiles solves the race condition.

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[jian-lin]

I tried to solve the same issue in #204290 and @ElvishJerricco wrote some very helpful advice in #204290 (comment). He may be interested in reviewing this.

[ElvishJerricco]

I don't think I like having one unit per user like this. Seems very cluttered.

I was looking at systemd-sysusers, hoping it had something for automatic home directory creation that we could use for inspiration, and the man page for sysusers.d says this:

systemd-sysusers only sets the home directory record in the user database. To actually create the directory, consider adding a corresponding tmpfiles.d(5) fragment.

I wonder if a tmpfiles thing is correct here? Sounds odd given the "tmp" in the name, but I think would get the job done.

[jsoo1]

I don't think I like having one unit per user like this. Seems very cluttered.

I was looking at systemd-sysusers, hoping it had something for automatic home directory creation that we could use for inspiration, and the man page for sysusers.d says this:

systemd-sysusers only sets the home directory record in the user database. To actually create the directory, consider adding a corresponding tmpfiles.d(5) fragment.

I wonder if a tmpfiles thing is correct here? Sounds odd given the "tmp" in the name, but I think would get the job done.

Ah right. tmpfiles was mentioned in the issue. I am not sure how that would be accomplished. Do you have an example to point me to?

edit: after some perusal of the manpage, I am also concerned about the "volatile" description of the files meant to be managed by tmpfiles.d.

[jsoo1]

... but it seems like tmpfiles.d should do the trick. Pushed the change @ElvishJerricco

[ElvishJerricco]

Well that's certainly much simpler :) Though, I'm intrigued by the ordering relationships on systemd-tmpfiles-setup.service.

DefaultDependencies=no
After=local-fs.target systemd-sysusers.service systemd-journald.service
Before=sysinit.target

So it will wait for all fstab FSes to be mounted, but is still early enough to be before sysinit.target. Are these semantics good enough for NixOS? I wonder if there's any services that also have DefaultDependencies=no and a user with a home directory that needs creation... I doubt it. A service really shouldn't assume that things like home directories are available before at least sysinit.target is reached, if not basic.target or even multi-user.target.

[jsoo1]

Well that's certainly much simpler :) Though, I'm intrigued by the ordering relationships on systemd-tmpfiles-setup.service.

DefaultDependencies=no
After=local-fs.target systemd-sysusers.service systemd-journald.service
Before=sysinit.target

So it will wait for all fstab FSes to be mounted, but is still early enough to be before sysinit.target. Are these semantics good enough for NixOS? I wonder if there's any services that also have DefaultDependencies=no and a user with a home directory that needs creation... I doubt it. A service really shouldn't assume that things like home directories are available before at least sysinit.target is reached, if not basic.target or even multi-user.target.

True. A quick grep also says that no service with DefaultDependencies = false and a user with home directories.

[jsoo1]

Another question: does local-fs.target complete after network file systems? What if a home directory is on nfs?

[ElvishJerricco]

That's great then! Assuming we can find some tests to pass (or make one), I like the look of this! Solves a noticeable problem that if there's a mountpoint between / and $HOME, the activation script doesn't create the directory correctly.

As a tangential addendum, I wonder if we could have the activation script generate a systemd-sysusers config file instead of doing what it does. I dunno if that would actually have much benefit, but it'd be interesting to look into.

[ElvishJerricco]

Another question: does local-fs.target complete after network file systems? What if a home directory is on nfs?

Oh that's an interesting question. It looks like remote-fs.target is ordered before only multi-user.target in man bootup.... and it looks like NixOS doesn't even honor that:

$ systemctl list-dependencies --before remote-fs.target
remote-fs.target
○ ├─libvirtd.service
● ├─systemd-user-sessions.service
○ ├─virtlxcd.service
○ ├─virtqemud.service
○ └─virtvboxd.service

[jsoo1]

Seems like nfs could be a problem: systemd/systemd#1959

[ElvishJerricco]

I think that specific issue is not exactly related, but I think there is a sort of issue. That seems to be more of an issue with automount home directories, which is another thing to consider.

systemd will take care of it in most cases, because of the implicit ordering of mount units and because it does create intermediate directories when needed, but here's the problem that could come up: systemd-tmpfiles-setup.service can run before a remote FS in the user's $HOME path is mounted. This can cause two things to happen.

1. If the remote FS is mounted at $HOME directly, systemd-tmpfiles-setup.service will just make that directory exist beforehand, which is fine, although homeMode won't be set correctly once the FS is mounted.
2. If the remote FS is a parent directory of $HOME, systemd-tmpfiles-setup.service will make the home directory in the mountpoint directory and not in the FS, and then when the remote FS is mounted, it won't have the home directory in it.

[jsoo1]

I think that specific issue is not exactly related, but I think there is a sort of issue. That seems to be more of an issue with automount home directories, which is another thing to consider.

systemd will take care of it in most cases, because of the implicit ordering of mount units and because it does create intermediate directories when needed, but here's the problem that could come up: systemd-tmpfiles-setup.service can run before a remote FS in the user's $HOME path is mounted. This can cause two things to happen.

1. If the remote FS is mounted at `$HOME` directly, `systemd-tmpfiles-setup.service` will just make that directory exist beforehand, which is fine, although `homeMode` won't be set correctly once the FS is mounted.

2. If the remote FS is a parent directory of `$HOME`, `systemd-tmpfiles-setup.service` will make the home directory _in the mountpoint directory_ and not in the FS, and then when the remote FS is mounted, it won't have the home directory in it.

I see... 2 is I think the original root cause of the issue and 1 also seems a problem, but less egregious.

Perhaps the oneshot services are the best way to go, here... what do you think?

New issues found

[ElvishJerricco]

FWIW, this is still objectively an improvement over the existing code. This problem also existed, but more severely, before this change.

[jsoo1]

Hm. Would oneshot services fix both the automount and nfs problems @ElvishJerricco ?

[ElvishJerricco]

Well, one thing that concerned me was that neither solution would recreate user home directories if they were deleted and then a system config was activated. So I added this to test it out, expecting for it to fail with this PR but succeed without it:

diff --git a/nixos/tests/mutable-users.nix b/nixos/tests/mutable-users.nix
index ebe32e6487e..d9025295934 100644
--- a/nixos/tests/mutable-users.nix
+++ b/nixos/tests/mutable-users.nix
@@ -69,5 +69,9 @@ import ./make-test-python.nix ({ pkgs, ...} : {
         for file in files_to_check:
             assert machine.succeed(f"sha256sum {file}") == expected_hashes[file]
             assert machine.succeed(f"stat {file}") == expected_stats[file]
+
+    with subtest("activate does change things"):
+        machine.succeed("/run/current-system/bin/switch-to-configuration test")
+        machine.succeed('test -e /home/dry-test') # home WAS recreated
   '';
 })

Above this diff, the test removes the home directory so that it can make sure dry-activate doesn't recreate it. To my surprise, it succeed even with this PR. Turns out switch-to-configuration has logic for tmpfiles which automatically covers it with the tmpfiles solution. I think we'd have to add a bunch of logic if we wanted to do this with oneshots.

Between that, the fact that creating a oneshot per user seems excessive, and the fact that this problem has always existed for users of remote-fs homes, I think the tmpfiles solution is probably the right way to go.

I would also recommend adding that diff to this PR. Once that's done I'll ask ofborg to tests mutableUsers and user-home-mode, and we'll be looking pretty good.

[jsoo1]

Adding one formatting commit along with your patch. Thank you very much @ElvishJerricco !

[ElvishJerricco]

@ofborg test mutableUsers user-home-mode

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/2018

[pennae]

this is incorrect. tmpfile paths do not respect shell escaping rules, in fact the escaping rules documented in the manpage are not even correct (systemd/systemd#26955). until that is sorted out we'd better be very careful with using tmpfiles to create arbitrary paths. user names themselves are fine, but there is no restriction on the parent directory. until systemd is fixed to actually follow its escaping rules we had better leave it as is.

[jsoo1]

Ok good call, thank you!

[jsoo1]

Is there a library function to c-escape a string that you know of, on that topic?

[pennae]

don't think so, almost certainly not by the rules systemd requires (tmpfiles also processes %<x> directives, so those need escaped as well)

[jsoo1]

I see. Well it's probably not impossible...

[ElvishJerricco]

Well crap. Lets hold off on this until we have a solution or at least an answer to that bug. I'll be looking into it.

tmpfiles bug could break things

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/users-users-name-createhome-not-creating-home-directory/30779/4

[CarlHMitchell]

Looks like that systemd bug is fixed at least as of Systemd v254. This PR merged & fixed it. Nixpkgs 23.11 includes that version of Systemd, so this should be unblocked.

[cafkafk]

mutable-users.nix and user-home-mounts.nix tests do pass, is there anything else here in need of fixup before being merged?

[cafkafk]

Should be moved to new release notes.

[ElvishJerricco]

@jsoo1 Can you rebase on unstable? Personally, I'd prefer to see the PR without the reformat to make it easier to review. I think there are plans to auto-format all of nixpkgs once the nixfmt RFC 166 lands.

@nikstur Is there anything about this PR that would conflict with or need adaptation for the merge of #270727 and its very similar approach to the home directory?

[nyabinary]

@jsoo1 Can you rebase on unstable? Personally, I'd prefer to see the PR without the reformat to make it easier to review. I think there are plans to auto-format all of nixpkgs once the nixfmt RFC 166 lands.

@nikstur Is there anything about this PR that would conflict with or need adaptation for the merge of #270727 and its very similar approach to the home directory?

Think it would be better to reopen clone this & rebase it and then open another PR since it doesn't seem the author is responsive,

[jsoo1]

Sorry I am listening but I don't have bandwidth for this. @ixmatus and @bacchanalia can you help please?

[nyabinary]

bump
cc @ixmatus @bacchanalia

[ixmatus]

It might be some time before I have the personal bandwidth to push this forward. If I have some free-time in $JOB to push this along I'll try, maybe @bacchanalia and I can try working on it together a bit.