Kubeadm DirAvailableCheck Should Ignore lost+found

[bagnaram]

BUG REPORT

DirAvailableCheck should ignore Linux lost+found` directories.

Versions

kubeadm version (use kubeadm version):

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration: CAPA
• OS CentOS Linux release 7.7.1908 (Core)
• Kernel Linux ip-10-0-0-33.us-west-1.compute.internal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Others:
  kubeadm Version:
  kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:01:11Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

What happened?

When mounting /var/lib/etcd on an external volume using a Linux filesysten it creates a lost+found directory be default. This is necessary for the fsck utility. However this causes the preflight checks to fail for etcd because the directory isn't empty.

What you expected to happen?

DirAvailableCheck should ignore lost+found

How to reproduce it (as minimally and precisely as possible)?

Mount /var/lib/etcd on a different volume with ext4 and run kubeadm init

Anything else we need to know?

[neolit123]

/assign @rosti @ereslibre

what do you think about this?

DirAvailableCheck should ignore Linux lost+found` directories.

i'm not familiar with lost+found

[CecileRobertMichon]

+1

I'm also running into this.

FWIW in my case the lost+found dir was added after mounting the etcd directory to an external data disk and was empty.

[fabriziopandini]

If the fix is small enough, IMO this is a candidate for backport

[neolit123]

When mounting /var/lib/etcd on an external volume using a Linux filesysten it creates a lost+found directory be default. This is necessary for the fsck utility. However this causes the preflight checks to fail for etcd because the directory isn't empty.

instead of mounting /var/lib/etcd to be the root of the external volume, why not mount the volume and sym-link a sub-folder to /var/lib/etcd? my understanding is that this solves the problem.

also kubeadm allows you to ignore preflight checks for DirAvailableCheck on per-directory granularity, except that Cluster API does not allow the kubeadm --ignore-preflight-errors flag (AFAIK?) and also it uses kubeadm v1beta1 that does not have the same functionality via config (v1beta2 does support that).

[bagnaram]

Although this can be avoided by using the directory hierarchy mentioned above, it doesn't seem to be a clean solution. Instead of checking for the existence of an empty directory during DirAvailableCheck what about only checking the contents of /var/lib/etcd that may collide or conflict during installation?

[neolit123]

granular conflict detection requires binding to directory structures that may be version specific or change in the future (e.g. etcd5 might change it's snapshot model completely).

if list+found is ignored by DirAvailableCheck this should apply to the check as a whole and for all directories that are checked with it.

[rosti]

In general, using the root directory of a particular volume is not a good idea. Furthermore, this is lost+found now. In the future we might discover some other similar instances (especially if this turns out to be a volume backed by Windows).

For now symlinking or bind mounting to a sub-directory should do the trick. You can also ignore the pre-flight check error.
We might try to specifically look for some etcd sub-dirs (such as member), but this is not very robust in my opinion.

[neolit123]

closing due to:

• we don't seem to agree that kubeadm should be aware of tooling specific folders
• ignoring the error is already possible with v1beta2