kubelet/dns issues when switching to coredns #3979
Comments
|
btw.: i am aware of the fact that this might be more of a bug in kubelet, because it's a bad idea to just copy the node's resolv.conf ( for the reasons above), but i guess kubespray needs to handle this somehow. (creating a sane resolv.conf for kubelet...) |
|
I have occur a issue like this, if i set the DNS value to the network interface. Even thought CoreDNS will be generate automate, But if restart the network, Old Value will be recover. TYPE="Ethernet" |
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
lifecycle/rotten
|
I'm having the same problem with centos 7 (I know, not great), the exact error log is: |
|
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
|
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@jqiuyin: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Is there any solution to this problem? |
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug
Environment:
1.12.4
Cloud provider or hardware configuration:
hw
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"):Linux 4.15.0-43-generic x86_64
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Version of Ansible (
ansible --version):Kubespray version (commit) (
git rev-parse --short HEAD):2.8.1
Network plugin used:
calico
Hi,
i think there is an issue regarding ubuntu 18.04 (or any other os which comes with systemd-resolved active), kubelet and coreDNS.
We upgraded our cluster from 10.11.x to 10.12.4 using the latest kubespray manifest. This changes dns from kubedns to coredns, which is ok so far.
Problem is that coredns resulted in a crashloop, because it detected a config loop. This came from the fact that kubelet copies the host's /etc/resolv.conf to the pods (including the coredns pod) and this resolv.conf contains a 127.0.0.53 entry coming from systemd-resolved, a dns cache, which is active by default in ubuntu 18.04.
Result is that coredns finds a resolve loop and other pods think that there's a resolver in their own pod ( aka 127.0.0.53) which isn't true.
So in order to run correctly, kubelet needs to either copy a dedicated upstream dns config, omitting the 127.0.0.53 entry or systemd-resolved needs to be switched of.
Switching off systemd-resolved (or dnsmasq... etc) isn't that great too, because kubelet itself ignores /etc/hosts for dns lookups and crashes after disabling the dns cache, because kubelet doesn't seem to handle the searchdomain config in resolv.conf correctly and then is unable to resolve it's own non-fqdn hostname (so it wont find something like k8s-node-01 - it works with a dns cache because these usually mix in /etc/hosts for their results...)
Bug should be reproducible with a fresh ubuntu 18.04 setup ( with standards enabled) and coredns enabled.
The text was updated successfully, but these errors were encountered: