[btrfs] Failed to create cluster with http://localhost:10248/healthz connection refused, timed out waiting for the condition #1945
Comments
|
Please check the known issues page in the docs. You can obtain more logs for us to debug with Can you share Basically from the logs so far we know API server didn't come up healthy. There are any number of possible causes. The only known kind bug is related to ZFS (should be fixed at HEAD) but there are known broken environments (e.g. the firewalld issue) |
BenTheElder
added
the
triage/needs-information
|
Will reopen when we have more details. As-is this is not actionable for ~one week now. |
|
I'm sorry for the delay. I looked at known problems but I don't see anything immediately obvious. The See 345396429.tar.gz. |
|
It looks like you're running on btrfs, I'm not surprised to see that isn't working. Kubernetes etc. are rarely used on these COW filesystems versus overlayfs on e.g. ext4. and indeed there's a rather large panic in the
|
BenTheElder
changed the title
|
We do make a best effort to support it, but I don't think any of the maintainers use btrfs / have a test env handy and Kubernetes CI definitely doesn't run on it. I thought this was sufficient, apparently not 😞 |
|
Actually, it looks like we're not passing You can manually configure this as a possible workaround while we work on fixing the check. kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraMounts:
- hostPath: /dev/mapper
containerPath: /dev/mapperlike: cat << EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraMounts:
- hostPath: /dev/mapper
containerPath: /dev/mapper
EOF |
|
Did you configure docker at all? It's really strange that you have storage driver = overlay2 while backing filesystem = btrfs. https://docs.docker.com/storage/storagedriver/select-storage-driver/#supported-backing-filesystems btrfs : btrfs is supported, overlay2/overlay support xfs, ext4. The overlay2 but actually btrfs is tripping up our check to mount /dev/mapper, but I'm surprised this is even a working docker environment. EDIT: relevant excerpt from |
|
I'm a bit confused. Are we talking about btrfs storage driver or something else? Because my rootfs is btrfs. I didn't mess with changing the default storage driver in docker. So /var/lib/docker is on a btrfs formatted partition. I guess that's what 'backing filesystem' means. I've been running docker containers like this for a long time and didn't have any issues before. Should I move it to ext4? I tried the provided config file and it seems to be working: Thanks for your help! |
So usually when docker is on If this is working fine we shouldn't make you change your environment config! We should improve kind to detect this case as also requiring |
|
My rootfs used to be ext4. I moved it over to btrfs a while back. I didn't realize I needed to change anything. |
|
Ohh interesting. This may have confused docker into using overlay on btrfs even though their docs don't show that to be supported. It seems it works though? So we should maybe still alter the check in kind for auto-mounting /dev/mapper 😅 |
|
Right, it was working fine for maybe a year except the issue with kind. I recreated /var/lib/docker and it seems to have switched to btrfs storage driver. Kind works now without the special config file. |
|
@BenTheElder Hi Ben, |
|
@Napoleon0621 please file a distinct issue in the future so we can keep bug tracking organized. If you're failing at a certain node count you likely just don't have enough resources to support this many. IOPs is one that frequently bites people that otherwise have plenty of ram and cpu. You can try putting etcd in memory (search the issue tracker) but eventually any machine will run out of IO and parts of Kubernetes dependencies are heavy on it. |
|
I was able to bring up 10 node cluster without any issues myself. |
|
/close
I think that the original problem was solved, we can open new ones to have the issues organised so users can look for the information or find it easily |
|
@aojea: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@aojea this is not really fixed, we need a more intelligent check for mounting mapper, even if docker doesn't officially list this as supported it seems to happen to multiple users AND continue working otherwise. |
|
is this related @BenTheElder ? |
|
@aojea the kubernetes PR could be an issue but the problem here is this #1945 (comment) we currently detect when to mount /dev/mapper based on docker's storage driver. we need to detect based on the backing filesystem as well (which is more annoying to parse it's like a list of status objects IIRC ...) |
|
this is an issue for all Fedora33 users , since btrfs is the default /assign |
What if I am installing on windows and Im not referring to a yaml ? How do I apply this fix ? I am installing via kind create cluster --name prometheus --image kindest/node:v1.18.4 and I am facing same error as reported [here(https://github.com//issues/2323) |
|
@ed87 your issue is most likely not this issue.
Please see:
v1.18.4 image is not from a remotely recent KIND release, seeing as we push the latest patch version, and the latest 1.18 patch release before upstream kubernetes stopped supporting it is 1.18.20 |
What happened:
make build && bin/kind create clusterWhat you expected to happen:
How to reproduce it (as minimally and precisely as possible):
git clone [email protected]:kubernetes-sigs/kind.git kind-repocd kind-repomake build && bin/kind create clusterAnything else we need to know?:
Environment:
kind version):kind v0.10.0-alpha+aebedcf8f4bf68 go1.15.5 linux/amd64kubectl version):v1.19.3docker info):19.03.13/etc/os-release):Gentoo(recently updated)The text was updated successfully, but these errors were encountered: