Login and sudo failures possibly due to pam_ccreds issue

[austinbutler]

Describe the bug

I get random failures when trying to log in or use sudo: sudo: PAM authentication error: Error in service module. It seems to happen most often on first login or first sudo use. Usually after 2-3 tries it succeeds. Sometimes during login it works first try.

To Reproduce

1. Configure NFS mount with Kerberos
2. Try to log in or sudo, sometimes after maybe 10s it fails with sudo: PAM authentication error: Error in service module

Expected behavior

Login and sudo always work when you provide the correct password.

Additional context

In the journal I see this:

Aug 09 11:16:56 austin-laptop sudo[6409]: pam_unix(sudo:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=  user=austin
Aug 09 11:17:14 austin-laptop sudo[6409]: pam_krb5(sudo:auth): authentication failure; logname=austin uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=
Aug 09 11:17:14 austin-laptop sudo[6409]: pam_ccreds: failed to open cached credentials "/var/cache/.security.db": No such file or directory
Aug 09 11:17:16 austin-laptop sudo[6409]:   austin : PAM authentication error: Error in service module ; TTY=pts/0 ; PWD=/home/austin ; USER=root ; COMMAND=/run/current-system/sw/bin/journalctl
Aug 09 11:17:22 austin-laptop sudo[6417]:   austin : TTY=pts/0 ; PWD=/home/austin ; USER=root ; COMMAND=/run/current-system/sw/bin/journalctl
Aug 09 11:17:22 austin-laptop sudo[6417]: pam_unix(sudo:session): session opened for user root by (uid=0)

I do have an NFS mount that uses krb5 for authentication, and I've added "Enterprise Login (Kerberos)" in Gnome Online Accounts. The NFS mount is set to only mount on access (x-systemd.automount) because the computer that is the kdc is not always online. To be clear, I log in to the system with a normal local account, Kerberos is just for the NFS mount.

My NFS mount from configuration.nix:

  fileSystems."/mnt/nas" = {
    device = "server:/mnt/data";
    fsType = "nfs";
    options = [
      "soft"
      "noauto"
      "noatime"
      "nodiratime"
      "x-systemd.automount"
      "x-systemd.mount-timeout=10"
      "timeo=14"
      "users"
    ];
  };

This is my Kerberos config from configuration.nix:

  krb5.enable = true;
  krb5.domain_realm = {
    ".mydomain" = "MY_REALM";
    "mydomain" = "MY_REALM";
  };
  krb5.realms = {
    "MY_REALM" = {
      admin_server = "admin_server";
      kdc = "admin_server";
      master_kdc = "admin_server";
    };
  };
  krb5.libdefaults = {
    default_realm = "MY_REALM";
    dns_lookup_kdc = false;
    dns_lookup_realm = false;
  };
  krb5.extraConfig = ''
    [logging]
      kdc          = SYSLOG:NOTICE
      admin_server = SYSLOG:NOTICE
      default      = SYSLOG:NOTICE
  '';

What stands out to me from the log is pam_ccreds: failed to open cached credentials "/var/cache/.security.db": No such file or directory. pam_ccreds is installed, but that file does not exist.

Notify maintainers

The pam module doesn't seem to have a maintainers list, so based on commit history pinging @Mic92 @infinisil @flokli.

Metadata

❯ nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/gq21dvkri5h3bkg2ly4787kjx8az1s8w-bash-interactive-4.4-p23-dev
copying path '/nix/store/gq21dvkri5h3bkg2ly4787kjx8az1s8w-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.7.12, NixOS, 20.09pre237606.b50ef9afa11 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(root): `"nixos-20.09pre237606.b50ef9afa11, home-manager"`
 - channels(austin): `"home-manager"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - pam_ccreds
# a list of nixos modules affected by the problem
module:
  - pam

[Mic92]

I never used pam_ccreds but it wants /var/cache/.security.db to be created, whatever this file do:

Aug 09 11:17:14 austin-laptop sudo[6409]: pam_ccreds: failed to open cached credentials "/var/cache/.security.db": No such file or directory

[volhovm]

I have the same problem: although my sudo works correctly when I enter the valid password, if I make a mistake it outputs sudo: PAM authentication error: Error in service module and exits, instead of the usual Sorry, try again. and asking me the password again twice. This issue only exist if I turn on Kerberos, that is just commenting out the following solves the issue:

  krb5 = {
    enable = true;
    libdefaults.default_realm = "MY.UNI.SERVER";
  };

What I see in the logs is this:

Oct 03 11:03:57 keshet sudo[27777]: pam_unix(sudo:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=volhovm rhost=  user=volhovm
Oct 03 11:03:57 keshet sudo[27777]: pam_krb5(sudo:auth): authentication failure; logname=volhovm uid=1000 euid=0 tty=/dev/pts/0 ruser=volhovm rhost=
Oct 03 11:03:57 keshet sudo[27777]: pam_ccreds: failed to open cached credentials "/var/cache/.security.db": No such file or directory
Oct 03 11:04:00 keshet sudo[27777]:  volhovm : PAM authentication error: Error in service module ; TTY=pts/0 ; PWD=/home/volhovm/blablabla ; USER=root ; COMMAND=/run/blablabla

I do not remember whether this has been an issue since I started to use Kerberos (almost a year ago), or later after some nixos update. I suppose it could be the latter.

[austinbutler]

Some further observations. I disabled the built-in pam.nix module and imported a modified pam.nix from local nixpkgs. The local one has the two pam_cccreds lines commented out, removing it from the equation. With this setup there's still a lengthy delay on incorrect password (~15s?).

Nov 22 13:13:52 austin-laptop sudo[11088]: pam_unix(sudo:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=  user=austin
Nov 22 13:14:10 austin-laptop sudo[11088]: pam_krb5(sudo:auth): authentication failure; logname=austin uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=
Nov 22 13:14:40 austin-laptop sudo[11088]: pam_krb5(sudo:auth): authentication failure; logname=austin uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=
Nov 22 13:15:10 austin-laptop sudo[11088]: pam_krb5(sudo:auth): authentication failure; logname=austin uid=1000 euid=0 tty=/dev/pts/0 ruser=austin rhost=

With pam_krb5 out of the way entirely the incorrect password delay is the normal ~2s.

[austinbutler]

For now I'm settling with just removing all krb5 mentions from pam.nix entirely. My sole use case for Kerberos is NFS encryption, logging in to the system should not consult Kerberos. I think NixOS assumes if you enable anything with Kerberos you want to use it for login as well. Whether that's the correct assumption, I don't know, I'm certainly not a Kerberos or PAM expert (not even close), and maybe my setup is pretty unique.

Given that, in my nixpkgs fork I've added a pamIntegration option to krb5, and this seems to work. Not sure whether this would warrant a PR or not...

let
  # https://github.com/NixOS/nixpkgs/issues/95017
  krbPamFix = (builtins.fetchTarball {
    url =
      "https://github.com/austinbutler/nixpkgs/archive/35b5964d4f87d487a757400c4648e670a4000d36.tar.gz";
    sha256 = "0yb4vdyjlp4q0qahma793f7a8wx4hvryb52qb42q24b0f94kq9zh";
  });
in {
  disabledModules = [ "config/krb5/default.nix" "security/pam.nix" ];
  imports = [
    ./hardware-configuration.nix
    <home-manager/nixos>
    "${krbPamFix}/nixos/modules/security/pam.nix"
    "${krbPamFix}/nixos/modules/config/krb5/default.nix"
  ];

  krb5.enable = true;
  krb5.pamIntegration = false;
  # Rest of config...
}

With krb5.enable = true and krb5.pamIntegration = true (or just omitting krb5.pamIntegration):

❯ rg krb5 /etc/pam.d/login
6:account sufficient /nix/store/06llkrpj4fjv1qwy0q0ja4wdixd8mg3l-pam-krb5-4.9/lib/security/pam_krb5.so
33:auth [default=ignore success=1 service_err=reset] /nix/store/06llkrpj4fjv1qwy0q0ja4wdixd8mg3l-pam-krb5-4.9/lib/security/pam_krb5.so use_first_pass
45:password sufficient /nix/store/06llkrpj4fjv1qwy0q0ja4wdixd8mg3l-pam-krb5-4.9/lib/security/pam_krb5.so use_first_pass
59:session optional /nix/store/06llkrpj4fjv1qwy0q0ja4wdixd8mg3l-pam-krb5-4.9/lib/security/pam_krb5.so

With krb5.enable = true and krb5.pamIntegration = false:

❯ rg krb5 /etc/pam.d/login

This doesn't address the apparent pam_ccreds issue, though. @volhovm guessing for you krb5 is actually used for login/sudo, not just NFS?

[austinbutler]

Interestingly during my experimenting today it apparently did create /var/cache/.security.db, so I wonder if it's some sort of ordering issue.

[stale]

I marked this as stale due to inactivity. → More info

[austinbutler]

Still an issue for me today.

[austinbutler]

This seems to be resolved by #184553 (set security.pam.krb5.enabled = false;).