unable to load kernel module "configs"

[mitar]

On Linux Ubuntu 18.04 I am getting the following error/warning:

[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.15.0-36-generic
DOCKER_VERSION: 17.03.2-ce
DOCKER_GRAPH_DRIVER: aufs
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
	[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module "configs": output - "modprobe: FATAL: Module configs not found in directory /lib/modules/4.15.0-36-generic\n", err - exit status 1

The reason is that the modules directory on the host looks like this:

/lib/modules/4.15.0-36-generic$ ls -al
total 5288
drwxr-xr-x  5 root root    4096 Oct  4 08:39 .
drwxr-xr-x  7 root root    4096 Oct  4 08:39 ..
lrwxrwxrwx  1 root root      40 Sep 24 07:08 build -> /usr/src/linux-headers-4.15.0-36-generic
drwxr-xr-x  2 root root    4096 Sep 24 07:08 initrd
drwxr-xr-x 15 root root    4096 Oct  4 08:39 kernel
-rw-r--r--  1 root root 1264027 Oct  4 08:39 modules.alias
-rw-r--r--  1 root root 1244585 Oct  4 08:39 modules.alias.bin
-rw-r--r--  1 root root    7719 Sep 24 07:08 modules.builtin
-rw-r--r--  1 root root    9770 Oct  4 08:39 modules.builtin.bin
-rw-r--r--  1 root root  552081 Oct  4 08:39 modules.dep
-rw-r--r--  1 root root  780354 Oct  4 08:39 modules.dep.bin
-rw-r--r--  1 root root     317 Oct  4 08:39 modules.devname
-rw-r--r--  1 root root  206127 Sep 24 07:08 modules.order
-rw-r--r--  1 root root     540 Oct  4 08:39 modules.softdep
-rw-r--r--  1 root root  589765 Oct  4 08:39 modules.symbols
-rw-r--r--  1 root root  719596 Oct  4 08:39 modules.symbols.bin
drwxr-xr-x  3 root root    4096 Oct  4 08:39 vdso

kind mounts modules directory into the Docker image, but not also build subdirectory.

[BenTheElder]

FWIW that warning should just be a warning, we intentionally ignore preflight complaints from kubeadm.

…

On Tue, Oct 9, 2018, 00:29 Mitar ***@***.***> wrote: On Linux Ubuntu 18.04 I am getting the following error/warning: [preflight] The system verification failed. Printing the output from the verification: KERNEL_VERSION: 4.15.0-36-generic DOCKER_VERSION: 17.03.2-ce DOCKER_GRAPH_DRIVER: aufs OS: Linux CGROUPS_CPU: enabled CGROUPS_CPUACCT: enabled CGROUPS_CPUSET: enabled CGROUPS_DEVICES: enabled CGROUPS_FREEZER: enabled CGROUPS_MEMORY: enabled [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module "configs": output - "modprobe: FATAL: Module configs not found in directory /lib/modules/4.15.0-36-generic\n", err - exit status 1 The reason is that the modules directory on the host looks like this: /lib/modules/4.15.0-36-generic$ ls -al total 5288 drwxr-xr-x 5 root root 4096 Oct 4 08:39 . drwxr-xr-x 7 root root 4096 Oct 4 08:39 .. lrwxrwxrwx 1 root root 40 Sep 24 07:08 build -> /usr/src/linux-headers-4.15.0-36-generic drwxr-xr-x 2 root root 4096 Sep 24 07:08 initrd drwxr-xr-x 15 root root 4096 Oct 4 08:39 kernel -rw-r--r-- 1 root root 1264027 Oct 4 08:39 modules.alias -rw-r--r-- 1 root root 1244585 Oct 4 08:39 modules.alias.bin -rw-r--r-- 1 root root 7719 Sep 24 07:08 modules.builtin -rw-r--r-- 1 root root 9770 Oct 4 08:39 modules.builtin.bin -rw-r--r-- 1 root root 552081 Oct 4 08:39 modules.dep -rw-r--r-- 1 root root 780354 Oct 4 08:39 modules.dep.bin -rw-r--r-- 1 root root 317 Oct 4 08:39 modules.devname -rw-r--r-- 1 root root 206127 Sep 24 07:08 modules.order -rw-r--r-- 1 root root 540 Oct 4 08:39 modules.softdep -rw-r--r-- 1 root root 589765 Oct 4 08:39 modules.symbols -rw-r--r-- 1 root root 719596 Oct 4 08:39 modules.symbols.bin drwxr-xr-x 3 root root 4096 Oct 4 08:39 vdso kind mounts modules directory into the Docker image, but not also build subdirectory. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#61>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq0Bk5XwRINSWRge36vNmdpxRetYVks5ujFBigaJpZM4XR3d1> .

[neolit123]

kind mounts modules directory into the Docker image, but not also build subdirectory.

is that the actual issue here?

unable to load kernel module "configs"

here is the validator code for reference:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/kernel_validator.go#L219

the kernel module that kubeadm cannot find can be used to obtain information of how the kernel was built while that same kernel is running. instead of a module it also looks for the configuration as a file written on disk. for me on Ubuntu 17.10 i can find it in the boot/ folder.

this is not a critical issue, but it's kind of odd that it's missing for you completely.

so where is the file located for you?
the contents should look like this:

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 4.13.0-41-generic Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
...

[mitar]

The .config file is located under /usr/src/linux-headers-4.15.0-36-generic/.config which is not mounted above (see the symlink). I am not sure where exactly is kube searching for this file though. But in my case it is under /lib/modules/4.15.0-36-generic/build/.config.

[BenTheElder]

I don't think a cluster per se needs this, kubeadm wants to read it to validate the kernel a bit.

[neolit123]

linux user space is fun and ambiguity.

these are unchecked:

/usr/src/linux-headers-4.15.0-36-generic/.config
/lib/modules/4.15.0-36-generic/build/.config

see:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/kernel_validator.go#L176-L182

I don't think a cluster per se needs this, kubeadm wants to read it to validate the kernel a bit.

not at all. in fact i'm seeing reports from Ubuntu 18.x users all the time and nobody reports this.
so @mitar i suggest you just copy the config to one of the above folders to silence the warning.

seems like something is wrong on your distro.

/close

[k8s-ci-robot]

@neolit123: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

linux user space is fun, ambiguity.

these are unchecked:

/usr/src/linux-headers-4.15.0-36-generic/.config
/lib/modules/4.15.0-36-generic/build/.config

see:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/kernel_validator.go#L176-L182

I don't think a cluster per se needs this, kubeadm wants to read it to validate the kernel a bit.

not at all. in fact i'm seeing reports from Ubuntu 18.x users all the time and nobody reports this.
so @mitar i suggest you just copy the config to one of the above folders to silence the warning.

seems like something is wrong on your distro.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mitar]

I would not really claim that it does not matter how this works on Ubuntu, because many people use it. Especially the new LTS version of it.

If this is not important, I would suggest this is silenced, or at least prefixed with a message that any potential following errors do not matter.

It should not be that a well-behaved program is issuing unnecessary warnings.

So which exactly files are checked? Which files are expected to be there?

[neolit123]

I would not really claim that it does not matter how this works on Ubuntu, because many people use it. Especially the new LTS version of it.

it already works for other including me, so there has to be something specific to your setup.

So which exactly files are checked? Which files are expected to be there?

the list of locations is defined here:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/system/kernel_validator.go#L176-L182

kubeadm cannot possibly handle all the different crazy distro flavors in terms of where they decide to store a certain kernel config file, but if you think yet another location can be added to the above list - PRs are welcome.

instead of doing that you can also just start the kernel module called configs.

[mitar]

I think you do not know how kind works. So kind runs kube inside a Docker container. For this kernel detection to work, it has to mount necessary stuff from the host. Because detection happens inside of the Docker container. Currently, it mounts /lib/modules into the Docker container, which is not even on the list you described for checking for the .config file. It is useful only to find the configs module. But Ubuntu seems not to ship with it in 18.04 LTS (I checked on default install).

On my host, the following file from that list is available, but kind does not map it into the container: "/boot/config-" + k.kernelRelease,.

[neolit123]

well in that case kind should probably just map the /boot/config... file to silence the warning.

[mitar]

I am glad that you agree. :-)

[BenTheElder]

Unfortunately you cannot actually silence these warnings, you can only tell kubeadm to ignore them AFAICT, which we do. We could not give kubeadm's output of course, but I think that might be somewhat less helpful.

I'll look more into mounting /boot for this particular issue.

[neolit123]

as we've discussed already the warning can be just ignored...

historically the validation code comes from node_e2e and were added by Random Liu.
then kubeadm became a consumer when preflight was introduced and the node_e2e switched to something else leaving kubeadm the only consumer. then the package moved to cmd/kubeadm.

i have some arguments against kubeadm needing to check for IKCONFIG sanity, so it seems like a stretched "nice-to-have" that originated from node_e2e. kubeadm probably shouldn't really care that much how the kernel on a node was built as long as the node is healthy and runs a kubelet.

[BenTheElder]

That sounds sane to me, shall we file an issue to discuss this upstream in kubeadm?

…

On Wed, Oct 10, 2018, 11:41 Lubomir I. Ivanov ***@***.***> wrote: as we've discussed already the warning can be just ignored... historically the validation code comes from node_e2e and were added by Random Liu. then kubeadm became a consumer when preflight was introduced and the node_e2e switched to something else leaving kubeadm the only consumer. then the package moved to cmd/kubeadm. i have some arguments against kubeadm needing to check for IKCONFIG sanity, so it seems like a stretched "nice-to-have" that originated from node_e2e. kubeadm probably shouldn't really care that much how the kernel on a node was built as long as the node is healthy and runs a kubelet. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#61 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4BqwqsrIrAwlVev2Oq5WEhvqdqKDdPks5ujj91gaJpZM4XR3d1> .

[BenTheElder]

We no longer hook the kubeadm output directly to stdout, so this is now benign + not spamming the output.