No DNS resolution with Kind running in rootful Podman #1939

BobyMCbobs · 2020-11-21T01:45:44Z

What happened:
I'm running Kind in rootful Podman and am having issues pulling images.
It appears that there's no DNS working.

What you expected to happen:

containers to be able to pull
DNS to resolve

How to reproduce it (as minimally and precisely as possible):

Bring up a Pod

$ kubectl run nginx --image nginx
pod/nginx created

Show Pod info

$ kubectl describe pod nginx
...
  Warning  Failed     4s    kubelet, kind-control-plane  Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head https://registry-1.docker.io/v2/library/nginx/manifests/latest: dial tcp: lookup registry-1.docker.io on 1.1.1.1:53: read udp 10.88.0.6:54866->1.1.1.1:53: read: no route to host
...

ping a domain from an existing Pod

$ kubectl -n local-path-storage exec -it deployment/local-path-provisioner -- ping dns.google
ping: bad address 'dns.google'
command terminated with exit code 1

ping an address from an existing Pod

$ kubectl -n local-path-storage exec -it deployment/local-path-provisioner -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=31.139 ms

So there appears to be internet, but no DNS.

Anything else we need to know?:

When running Podman as root, it appears to have DNS resolution

$ sudo podman run -it --rm alpine:3.12 ping dns.google
PING dns.google (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: seq=0 ttl=42 time=31.008 ms

Environment:

kind version: (use kind version): kind v0.9.0 go1.15.2 linux/amd64
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Podman version: (use podman info):

version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Thu Oct  8 05:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

OS (e.g. from /etc/os-release):

NAME=Fedora
VERSION="33.20201030.0 (Silverblue)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33.20201030.0 (Silverblue)"
...

The text was updated successfully, but these errors were encountered:

BobyMCbobs · 2020-11-21T01:46:51Z

/assign @aojea

aojea · 2020-11-21T23:01:16Z

I can' t reproduce it in my Fedora 32 and podman podman 2.1.1.

kind]$ sudo kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          40s   10.244.0.5   kind-control-plane   <none>           <none>

usual suspects are firewalld ot iptables mode
https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

BobyMCbobs · 2020-11-22T01:21:50Z

@aojea thank you for following up!

I installed a fresh copy of Fedora Silverblue 33 in Boxes.
Downloaded kind (applied a setuid for root on it) and kubectl.
Brought up the cluster and went through the original steps in this ticket but was unable to replicate my issue.
I haven't tweaked any system settings.

It appears to be using nftables also:

$ sudo cat /etc/firewalld/firewalld.conf | grep Firewall
# FirewallBackend
FirewallBackend=nftables

BobyMCbobs · 2020-11-22T06:10:56Z

OK I found the issue @aojea.

There's an interference with my firewalld config.
After systemctl disable --now firewalld, things appear to be working now:

kubectl get pods -A -w
NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
default              nginx                                        0/1     ContainerCreating   0          1s
kube-system          coredns-f9fd979d6-54xxz                      1/1     Running             0          75s
kube-system          coredns-f9fd979d6-pw88w                      1/1     Running             0          75s
kube-system          etcd-kind-control-plane                      1/1     Running             0          79s
kube-system          kindnet-7k7kd                                1/1     Running             0          75s
kube-system          kube-apiserver-kind-control-plane            1/1     Running             0          79s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running             0          78s
kube-system          kube-proxy-ltfnn                             1/1     Running             0          75s
kube-system          kube-scheduler-kind-control-plane            1/1     Running             0          78s
local-path-storage   local-path-provisioner-78776bfc44-pfwnj      1/1     Running             0          75s
default              nginx                                        1/1     Running             0          12s

I'll review my firewalld configuration.

BenTheElder · 2020-11-22T07:46:53Z

Sounds like https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld but with podman as well

aojea · 2020-11-22T09:52:11Z

/close
Thank for verifying ... iptables and DNS... it's always iptables and DNS ... 🤣

k8s-ci-robot · 2020-11-22T09:52:18Z

@aojea: Closing this issue.

In response to this:

/close
Thank for verifying ... iptables and DNS... it's always iptables and DNS ... 🤣

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BobyMCbobs · 2020-11-22T19:36:57Z

Sounds like https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld but with podman as well

@BenTheElder

Thank for verifying ... iptables and DNS... it's always iptables and DNS ...

@aojea

Thank you for following up, I wasn't sure that this was a duplicate issue - sorry about that.

I switched to iptables as per the known-issues page. Working fine now!

BenTheElder · 2020-11-30T20:12:53Z

We might need to update the known issues page.

I'm pretty certain I have iptables-nft working on my corp workstation, but I'm not sure if firewalld etc. are involved actually. (it's dockerd on ~debian).

aojea · 2020-12-01T07:40:38Z

hi @BobyMCbobs , I just found your blog https://blog.calebwoodbine.com/kind-on-fedora-33/ and KIND should be independent of the iptables backend, it should work for iptables-nft (nftables) and iptables-legacy,.
Previous issues were with docker, because it is incompatible with firewalld as you can see in the known issues

https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

On Fedora 32 firewalld moved to nftables backend by default. This seems to be incompatible with Docker, leading to KIND cluster nodes not being able to reach each other.

But if you say that you are running with podman I need to go deeper to understand better the problem
/reopen
/assign

@BobyMCbobs do you still have an environment to reproduce this?
if affirmative, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

k8s-ci-robot · 2020-12-01T07:40:45Z

@aojea: Reopened this issue.

In response to this:

hi @BobyMCbobs , I just found your blog https://blog.calebwoodbine.com/kind-on-fedora-33/ and I KIND should be independent of the iptables backend, it should work for iptables-nft (nftables) and iptables-legacy,.
Previous issues were with docker, because it is incompatible with firewalld as you can see in the known issues

https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

On Fedora 32 firewalld moved to nftables backend by default. This seems to be incompatible with Docker, leading to KIND cluster nodes not being able to reach each other.

But if you say that you are running with podman I need to go deeper to understand better the problem
/reopen
/assign

@BobyMCbobs do you still have an environment to reproduce this?
if affirmtive, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BobyMCbobs · 2020-12-02T08:44:14Z

@BobyMCbobs do you still have an environment to reproduce this?
if affirmative, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

@aojea,
Thank you for following up.

I switched back to nftables for this output and made sure the same errors occurred (including the same image pull cause of DNS).

iptables-legacy-save output:

# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*nat
:PREROUTING ACCEPT [174:24468]
:INPUT ACCEPT [6:1262]
:OUTPUT ACCEPT [254:16284]
:POSTROUTING ACCEPT [254:16284]
:CNI-DN-dc6a4d127af830bf71995 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:CNI-dc6a4d127af830bf71995c2e - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -s 10.88.0.21/32 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j CNI-dc6a4d127af830bf71995c2e
-A CNI-DN-dc6a4d127af830bf71995 -s 10.88.0.0/16 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-dc6a4d127af830bf71995 -s 127.0.0.1/32 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-dc6a4d127af830bf71995 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j DNAT --to-destination 10.88.0.21:6443
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -m multiport --dports 40603 -j CNI-DN-dc6a4d127af830bf71995
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A CNI-dc6a4d127af830bf71995c2e -d 10.88.0.0/16 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j ACCEPT
-A CNI-dc6a4d127af830bf71995c2e ! -d 224.0.0.0/4 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j MASQUERADE
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*mangle
:PREROUTING ACCEPT [1026:291610]
:INPUT ACCEPT [858:268404]
:FORWARD ACCEPT [131:7001]
:OUTPUT ACCEPT [1014:125876]
:POSTROUTING ACCEPT [1086:132101]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*raw
:PREROUTING ACCEPT [1026:291610]
:OUTPUT ACCEPT [1014:125876]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*security
:INPUT ACCEPT [838:267436]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1014:125876]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*filter
:INPUT ACCEPT [794:259911]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [944:118347]
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin admin overrides" -j CNI-ADMIN
-A CNI-FORWARD -d 10.88.0.21/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 10.88.0.21/32 -j ACCEPT
COMMIT
# Completed on Wed Dec  2 21:39:32 2020

Finally iptables-nft-save output outputs:

Unknown arguments found on commandline

A fresh Fedora Silverblue 33 ISO will render the same results to my testing. Set up is same instructions as my post and the steps in this issue.

aojea · 2020-12-02T08:48:54Z

@dcbw @squeed it seems that the cni plugins are using the legacy iptables instead of the nft version, that is the same one used by firewalld, causing the environment to fail ^^^
Do the CNI plugins check the iptables version to use? how can we sync it with firewalld?

squeed · 2020-12-02T09:51:22Z

Hmm. CNI binaries just use whatever /bin/iptables they find - how are you packaging them?

BenTheElder · 2020-12-02T09:58:23Z

does firewalld use iptables-nft? or just nft directly? kind is doing ~the same update-alternatives trick from kube-proxy, counting the rules in iptables and iptables-nft and selecting the one with the most rules. *docker* injects rules inside the container using the host iptables. podman I have no idea, but I'm guessing it doesn't do that, so we don't have anything to go on and default to legacy.

…

On Wed, Dec 2, 2020 at 1:51 AM Casey Callendrello ***@***.***> wrote: Hmm. CNI binaries just use whatever /bin/iptables they find - how are you packaging them? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1939 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK3PEUJQWPHXPY5B2HLSSYE3HANCNFSM4T5OXRJA> .

aojea · 2020-12-02T11:08:12Z

Hmm. CNI binaries just use whatever /bin/iptables they find - how are you packaging them?

They CNI plugin package come from fedora repos, podman just use them for the network iirc

BobyMCbobs · 2020-12-02T18:50:51Z

does firewalld use iptables-nft? or just nft directly?

@BenTheElder, here's the FirewallBackend section from my /etc/firewalld/firewalld.conf file

# FirewallBackend
# Selects the firewall backend implementation.
# Choices are:
#	- nftables (default)
#	- iptables (iptables, ip6tables, ebtables and ipset)
# FirewallBackend=nftables
FirewallBackend=iptables

However, there isn't an nftables program installed - so I believe it's using iptables-nft since it's installed

BenTheElder · 2020-12-03T02:45:46Z

Discussed more with Antonio -- There's two potential issues here:

We need everything on the host to line up and use iptables-legacy or iptables-nft (this is hopefully "just" a configuration issue on your machine)
Once the host is lined up, the node tries to detect which to use, but podman isn't writing any rules (reasonably) so we don't currently detect what the host is using and default to legacy. It's not obvious how we'd best solve that one even once the first one is correct

I'd been referring to the latter mostly, but I think you all are discussing the former 😅

squeed · 2020-12-07T11:35:07Z

This is, sadly, a common battle in this world, since there is no kernel-level flag that indicates which mode of iptables is in use. The "official" API is: where does /sbin/iptables symlink.

When we need to run iptables in a container, we generally bind-mount / to /host and then chroot. We then patch things up in the dockerfile with a simple script.

For example, see https://github.com/openshift/sdn/blob/master/images/iptables-scripts/iptables

BenTheElder · 2020-12-07T21:21:43Z

Interesting approach, thanks. I'm not sure how comfortable I'd be having kind always bind mount / into the container ... 🤔 (versus kube-proxy which is actually supposed to manage host network anyhow).

The kubernetes approach (which we currently use) is pretty nice, but requires some existing rules of some sort in the namespace unfortunately 😞

squeed · 2020-12-07T21:55:47Z

I just realized this is all moot - we're talking about CNI plugins executed by podman on the real node to bring up the kind "node" containers. Not the CNI executed by containerd / cri-o inside the kind nodes.

Scanning the issue, I suspect it's because we have some bits inside the CNI plugins that tweak iptables, and the node in question has firewalld with raw nftables. We support iptables-nft, but not nft itself.

BenTheElder · 2020-12-07T22:27:17Z

Scanning the issue, I suspect it's because we have some bits inside the CNI plugins that tweak iptables, and the node in question has firewalld with raw nftables. We support iptables-nft, but not nft itself.

Fedora seems to be pushing for firewalld / nft to be the default going forward, is there any hope for fixing this?

As is we point users to:
https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld
(should probably be renamed "firewalld nft" ...)

squeed · 2020-12-08T09:27:32Z

Fedora seems to be pushing for firewalld / nft to be the default going forward, is there any hope for fixing this?

Definitely! We probably just need to teak the "default" CNI configuration used by podman.

BenTheElder · 2021-02-28T18:54:13Z

@aojea came up again in #2099 I believe.

BenTheElder · 2021-03-24T04:30:23Z

@aojea is there something we can do here? Or should we just improve the known issues documentation and close this?

aojea · 2021-03-24T08:36:00Z

/close
I think is a distro problem, Fedora should align the components, firewalld, podman, cnis ,... they should use the same tools, I'm not in favor of workaround so "it just work in KIND" , because all the other container will be broken too

k8s-ci-robot · 2021-03-24T08:36:08Z

@aojea: Closing this issue.

In response to this:

/close
I think is a distro problem, Fedora should align the components, firewalld, podman, cnis ,... they should use the same tools, I'm not in favor of workaround so "it just work in KIND" , because all the other container will be broken too

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2021-05-28T00:21:45Z

I think is a distro problem, Fedora should align the components, firewalld, podman, cnis ,... they should use the same tools, I'm not in favor of workaround so "it just work in KIND" , because all the other container will be broken too

most other containers are not touching iptables though ..?

BobyMCbobs added the kind/bug Categorizes issue or PR as related to a bug. label Nov 21, 2020

k8s-ci-robot assigned aojea Nov 21, 2020

BenTheElder added the area/provider/podman Issues or PRs related to podman label Nov 21, 2020

k8s-ci-robot closed this as completed Nov 22, 2020

k8s-ci-robot reopened this Dec 1, 2020

aojea mentioned this issue Dec 1, 2020

bump go-iptables version containers/dnsname#43

Merged

BenTheElder mentioned this issue Feb 28, 2021

Kind onable to join worker nodes on RHEL 8.3 with podman #2099

Closed

BenTheElder added the kind/external upstream bugs label Feb 28, 2021

k8s-ci-robot closed this as completed Mar 24, 2021

BenTheElder mentioned this issue Apr 26, 2021

Error Starting Control Plane w/ Podman on Fedora 33 #2213

Closed

BenTheElder mentioned this issue May 28, 2021

Error rootless-cni-infra: modprobe: can't change directory to '/lib/modules': No such file or director #2271

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No DNS resolution with Kind running in rootful Podman #1939

No DNS resolution with Kind running in rootful Podman #1939

BobyMCbobs commented Nov 21, 2020

BobyMCbobs commented Nov 21, 2020

aojea commented Nov 21, 2020

BobyMCbobs commented Nov 22, 2020 •

edited

Loading

BobyMCbobs commented Nov 22, 2020

BenTheElder commented Nov 22, 2020

aojea commented Nov 22, 2020

k8s-ci-robot commented Nov 22, 2020

BobyMCbobs commented Nov 22, 2020

BenTheElder commented Nov 30, 2020

aojea commented Dec 1, 2020 •

edited

Loading

k8s-ci-robot commented Dec 1, 2020

BobyMCbobs commented Dec 2, 2020 •

edited

Loading

aojea commented Dec 2, 2020

squeed commented Dec 2, 2020

BenTheElder commented Dec 2, 2020 via email

aojea commented Dec 2, 2020

BobyMCbobs commented Dec 2, 2020

BenTheElder commented Dec 3, 2020

squeed commented Dec 7, 2020

BenTheElder commented Dec 7, 2020 •

edited

Loading

squeed commented Dec 7, 2020

BenTheElder commented Dec 7, 2020

squeed commented Dec 8, 2020

BenTheElder commented Feb 28, 2021

BenTheElder commented Mar 24, 2021

aojea commented Mar 24, 2021

k8s-ci-robot commented Mar 24, 2021

BenTheElder commented May 28, 2021

No DNS resolution with Kind running in rootful Podman #1939

No DNS resolution with Kind running in rootful Podman #1939

Comments

BobyMCbobs commented Nov 21, 2020

BobyMCbobs commented Nov 21, 2020

aojea commented Nov 21, 2020

BobyMCbobs commented Nov 22, 2020 • edited Loading

BobyMCbobs commented Nov 22, 2020

BenTheElder commented Nov 22, 2020

aojea commented Nov 22, 2020

k8s-ci-robot commented Nov 22, 2020

BobyMCbobs commented Nov 22, 2020

BenTheElder commented Nov 30, 2020

aojea commented Dec 1, 2020 • edited Loading

k8s-ci-robot commented Dec 1, 2020

BobyMCbobs commented Dec 2, 2020 • edited Loading

aojea commented Dec 2, 2020

squeed commented Dec 2, 2020

BenTheElder commented Dec 2, 2020 via email

aojea commented Dec 2, 2020

BobyMCbobs commented Dec 2, 2020

BenTheElder commented Dec 3, 2020

squeed commented Dec 7, 2020

BenTheElder commented Dec 7, 2020 • edited Loading

squeed commented Dec 7, 2020

BenTheElder commented Dec 7, 2020

squeed commented Dec 8, 2020

BenTheElder commented Feb 28, 2021

BenTheElder commented Mar 24, 2021

aojea commented Mar 24, 2021

k8s-ci-robot commented Mar 24, 2021

BenTheElder commented May 28, 2021

BobyMCbobs commented Nov 22, 2020 •

edited

Loading

aojea commented Dec 1, 2020 •

edited

Loading

BobyMCbobs commented Dec 2, 2020 •

edited

Loading

BenTheElder commented Dec 7, 2020 •

edited

Loading