[zfs] Failed to create cluster #2163
Comments
|
Came here looking since I have the exact same issue. Using almost the same software versions (only slightly older kubectl and Debian testing instead of Ubuntu and cgroup2 mounted) and zfs storage driver. I have no real evidence but I suspect it may have something to do with zfs. On a vm with all same versions but just no zfs it works fine. |
|
Can you both try kind at HEAD? ( We've made a number of patches around filesystems like ZFS coming in the next release (soon) e.g. #2149 Also worth noting though: zfs is not well supported in the container space, nobody is actively testing on this. I generally recommend sticking to something more common like ext4 for the docker data root at least (though zfs is super cool :-)). |
|
Thanks @BenTheElder :) Edit : also found this in journal.log I am using zfs because it sure is cool :) and is listed as a supported docker filesystem but I wasn't aware it is not (well) tested. Didn't run into issues before I encountered this with kind. I'll keep an eye on the commits for kind and probably move /var/lib/docker back to some other filesystem in the next days. |
|
I think docker works ok with ZFS, but I don't think they're testing it much. containerd, runc, kubernetes, etc. I'm actually more certain are not tested and in general pretty much the entire ecosystem is running overlayfs in production (or now bleeding edge rootless stuff with the fuse version). The overlay driver works on ext4 and xfs officially, but AFAIK may also work on other filesystems (but I don't think on zfs) https://docs.docker.com/storage/storagedriver/select-storage-driver/ (it's also documented as the preferred driver). In any case we try to support everything but non recommended docker configurations are best effort since we have a lot of other things to work on 😅 containerd/cgroups#83 suggests that the error you have now may actually be improper cgroups mounts, but it's difficult to tell from what we can see here. |
|
@devopstales short of running kind at HEAD if you could run with So far all I can see is that the API server never became healthy, which has a number of potential causes, generally broken nested container runtime in the host environment. |
BenTheElder
added
the
triage/needs-information
|
I didn't tried with kind at HEAD yet but This post solved my problem. #1719 (comment) |
|
Just tried the solution @devopstales mentioned above here and this works for me too :) |
|
Build and tested with kind at HEAD kind v0.11.0-alpha+8fe8b962521db0 go1.16.2 linux/amd64. Log below |
|
It sounds like the ZFS detection @ HEAD is not correct despite 69b84ab #1719 (comment) is supposed to be automatic now. from the logs in #2163 (comment) (thanks @devopstales!) I can see that the mapper mount is present, so it seems it must be this part not working ... kind/images/base/files/usr/local/bin/entrypoint Lines 79 to 81 in 8fe8b96 |
|
Just for completeness I've uploaded my logs here. Maybe it helps to have multiple logs. |
|
Sorry I'm going to be unavailable next week and been working on wrapping up some other things this week. I would guess at HEAD that since this is needed: the snippet of the entrypoint linked above is not working for some reason. but it will be difficult to diagnose without testing the behavior of each aspect on a real zfs system. I'm not sure if its the |
|
We still need to know if the filesystem check works or not. Basically the output of |
|
just brought up a vm looking now |
|
so looks like detection is working: |
|
but the change isn't reflected in crictl info. |
|
@mauilion what commit are you using? |
|
|
oh just read back 1 sec |
|
Files are created correctly on the host |
|
with kind@master and the 1.20.2 default image I still see detection working but like in the older broken way. I am building a new image with kind@master to see if it's fixed there. |
|
same problem with kind@master and a fresh build. Digging deeper. If I move the snapshotter param like up to the then I see crictl pick it up. working up a PR now. |
|
hmm I hoped #2131 would resolve that |
|
it really should. Maybe kind at master is still getting that old config.toml template or something? cause the old base-image? |
|
yeah that was it. I build a new base with That new image works well 🎉 So I think we just need to update the base image on master. |
|
@nkhine and @devopstales if you want to try this I pushed the |
|
@mauilion , perfect, thank you |
|
@BenTheElder just need a |
|
excellent, we're bumping again anyhow in #2207 |
|
I think this should have been fixed in v0.11.0 / v0.11.1? 😅 #2163 (comment) / the issue @mauilion identified (thank you!!!) were included in these releases. will re-open if we find out otherwise https://github.com/kubernetes-sigs/kind/releases/tag/v0.11.1 |
possibly a fix for kubernetes-sigs#2163 so that the `snapshotter = "native"` config workaround is not needed.
What happened:
cluster failed to start
What you expected to happen:
a cluster to start
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kind version):kubectl version):docker info):/etc/os-release):The text was updated successfully, but these errors were encountered: