"FATA [0000] permission denied" when /var is mounted noexec

[paramjeet01]

Environmental Info:
K3s Version: v1.26.0+k3s1

Node(s) CPU architecture, OS, and Version:
Linux hostgamma 4.18.0-425.13.1.el8_7.x86_64 #1 SMP Thu Feb 2 13:01:45 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
2 servers 2 agents

I'm trying to run k3s on redhat 8 server and i see this FATA [0000] permission denied whenever i try to run k3s server
But k3s command works and shows the help window.

Steps To Reproduce:

• Installed K3s:
  prefer-bundled-bin: true
  k3s_use_unsupported_config: true

Expected behavior:
Run k3s server
Actual behavior:
Throws FATA [0000] permission denied

Additional context / logs:
I can't get any logs other than this

[brandond]

1. How did you install k3s?
2. Are you running it as root? If not, see the section of the docs on running k3s rootless.
3. What is k3s_use_unsupported_config and where did you get it from?
4. Why are you using 1.26.0? There are much newer patch releases available on that minor.

[paramjeet01]

1. I'm using pyratlab/k3s ansible role , It works on other servers.
2. yes , running it as root with sudoers
3. It's a variable for pyratlabs/k3s and we're facing few issues earlier in other server so , we use this variable to resolve it
4. I'll try newer version and get back

[brandond]

I suggest you strace the command and figure out what it's trying to do that is getting a permission denied error. There's not enough context here to figure out what it's doing. Is this maybe a selinux problem?

[paramjeet01]

I tried the newer version : v1.27.1+k3s1 , Now it throws me this error message : k3s-server executable file not found in $path.
I have disabled the selinux for testing this.

[brandond]

is something preventing it from extracting the binaries to /var/lib/k3s/data/<RELEASE-HASH>, or running executables out of that directory? Do you have /var mounted readonly or noexec? You should have:

brandond@seago:~$ ls -la /var/lib/rancher/k3s/data/current/bin/k3s*
-rwxr-xr-x 1 root root 136943800 Apr 19 12:36 /var/lib/rancher/k3s/data/current/bin/k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-agent -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-certificate -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-completion -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-etcd-snapshot -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-secrets-encrypt -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-server -> k3s
lrwxrwxrwx 1 root root         3 Apr 19 17:00 /var/lib/rancher/k3s/data/current/bin/k3s-token -> k3s

You can try deleting /var/lib/rancher/k3s/data and it should re-extract everything, assuming whatever previously prevented it from extracting the files has been addressed.

I have disabled the selinux for testing this.

Did you previously have it enabled? Have you installed the k3s-selinux package?

[paramjeet01]

Yes , /var is mounted as noexec . And this is the strace output for the k3s v1.26.4+k3s1

newfstatat(AT_FDCWD, "/var/lib/rancher/k3s/data/24a53467e274f21ca27cec302d5fbd58e7176daf0a47a2c9ce032ee877e0979a/bin/k3s-server", {st_mode=S_IFREG|0755, st_size=132960592, ...}, 0) = 0
execve("/var/lib/rancher/k3s/data/24a53467e274f21ca27cec302d5fbd58e7176daf0a47a2c9ce032ee877e0979a/bin/k3s-server", ["/usr/local/bin/k3s", "server"], 0xc000198a90 /* 24 vars */) = -1 EACCES (Permission denied)
ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(2, "\33[31mFATA\33[0m[0000] permission d"..., 66FATA[0000] permission denied                            
) = 66
exit_group(1)                           = ?
+++ exited with 1 +++

yes , selinux was enabled at the start and disabled it later , then reinstalled the k3s .
Does k3s run executables from /var directory ? then , i can try mounting the /var without noexec or use --data-dir to change it's directory right ?

[brandond]

Does k3s run executables from /var directory?

Yep.

i can try mounting the /var without noexec

This is probably the easiest thing to do. I'm not sure which distros are now coming with /var noexec by default, but k3s will not work with this when using the default data-dir.

or use --data-dir to change it's directory

If you do this, set it in the config.yaml so that you don't have to remember to pass it on the command-line all the time.

[brandond]

Opened a PR to improve the error message.

[paramjeet01]

Thanks man , I'll check this configuration.

[brandond]

Reopening to track error message improvement PR.

[asunekants]

This is probably the easiest thing to do. I'm not sure which distros are now coming with /var noexec by default, but k3s will not work with this when using the default data-dir.

It's not a distro default, but rather a condition of the "CIS Red Hat Enterprise Linux 8 Benchmark" system hardening policy, which is not an uncommon policy in environments with regulatory or policy constraints;
The rationale for this particular setting is given in the benchmark as "Since the /var filesystem is only intended for variable files such as logs, set this option to ensure that users cannot run executable binaries from /var.", which is reasonable.

"k3s will not work with this when using the default data-dir", in combination with "Using a custom --data-dir under SELinux is not supported." (from [https://docs.k3s.io/advanced#enabling-selinux-enforcement]) creates a Catch-22 situation for running k3s on SCAP-hardened systems, it seems, as enforcing selinux is also a condition in the aforementioned benchmark;
I'd raise this as a bug.

[est-suse]

Validated on branch with commit / version

Validated on Master Branch:  2b24c9917cdfec92439ac68dd6706fffe20195a5

k3s version v1.27.1+k3s-2b24c991 (2b24c991)
go version go1.20.3
 

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-43-6 5.15.0-1033-aws #37~20.04.1-Ubuntu SMP Fri Mar 17 11:39:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
token: test
prefer-bundled-bin: true
k3s_use_unsupported_config: true
cluster-init: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Add a tmpfs mount to /var in fstab with noexec: by adding the following line: vi /etc/fstab
   tmpfs /var tmpfs defaults,nodev,nosuid,noexec 0 0

reboot the VM
4. Install k3s: curl https://get.k3s.io | INSTALL_K3S_COMMIT=2b24c9917cdfec92439ac68dd6706fffe20195a5 sh -s - server
5. Check the logs for the new error msg

Validation Results:

• k3s version used for validation:

Replicated on v1.27.1:

May 15 20:21:18 ip-172-31-33-29 k3s[2878]: time="2023-05-15T20:21:18Z" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
May 15 20:21:23 ip-172-31-33-29 k3s[2886]: time="2023-05-15T20:21:23Z" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
May 15 20:21:28 ip-172-31-33-29 k3s[2894]: time="2023-05-15T20:21:28Z" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"


Validated the new error message: 

May 15 22:48:33 ip-172-31-43-6 k3s[18528]: time="2023-05-15T22:48:33Z" level=fatal msg="exec /var/lib/rancher/k3s/data/3a5a046ca7504505788f8ab75ca8189219b088c6700b1e19a7aa5e755eb5bde9/bin/k3s-server failed: permission denied"