Docker CE (Not Docker Desktop) containers can not connect to its nameserver. #47

tfl-takashima · 2021-10-12T04:30:39Z

Hi,

Thanks for wsl-vpnkit, Docker containers of Docker CE (not docker desktop) on wsl2 can connect to external hosts by specifying ip address under VPN(PANGP) environment. But, it seems not be able to resolve hostname so the following curl command fails on the containers.

$ docker exec  -it hoge2 curl --head www.google.com
curl: (6) Could not resolve host: www.google.com; Unknown error

I checked the nameserver on docker containers are set to the address of virtual ethernet adapter for WSL and confirmed ping command to the address done successfully. And I can make the dns name resolution success when I disable the VPN or tweak metrics value of network adapters. Attaching containers to docker host (not bridge) network seems work as well.

As far as appearances go, only the access to the WSL virtual adapter per se for name resolution fails still using wsl-vpnkit. Is it an expected behavior?

Regards,

The text was updated successfully, but these errors were encountered:

sakai135 · 2021-10-12T06:03:15Z

I think I was able to recreate your issue locally, and #48 should fix the issue. Can you try the fix with the artifact in this build and verify that it works for you as well?

tfl-takashima · 2021-10-12T06:14:48Z

Hi @sakai135 ,

I really appreciate your kindness.
The build works fine 💯

Thanks again,

tfl-takashima · 2021-10-14T06:50:54Z

Hi @sakai135

I'm sorry but the issue occurs again. the docker container can not query the default nameserver again, and HTTP request won't success as follows:

$ docker run -it --rm busybox wget -S --spider www.google.com
wget: bad address 'www.google.com'
$ docker run -it --rm --net host busybox wget -S --spider www.google.com # <=== request success only when "host" network 
Connecting to www.google.com (142.250.206.228:80)
  HTTP/1.1 200 OK
  Date: Thu, 14 Oct 2021 06:37:56 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Server: gws
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: 1P_JAR=2021-10-14-06; expires=Sat, 13-Nov-2021 06:37:56 GMT; path=/; domain=.google.com; Secure
  Set-Cookie: NID=511=Wjc5oRPmlHYnVIWnQtGyO8rNfpgcEhMcmOehgOxJKYZDo8X3J9wHvR2gcNj6Mse_HJAXte950KN2-rTu0bw8P2ZSVDPtvVIGBDmEWe0euLPS1teEFfWryvmzf-p_XjTzMOUAjxZS3Ig-9_qL4rMwBRDoK3Ocrv8lYV3oUDauqC4; expires=Fri, 15-Apr-2022 06:37:56 GMT; path=/; domain=.google.com; HttpOnly
  Accept-Ranges: none
  Vary: Accept-Encoding
  Connection: close
  Transfer-Encoding: chunked

remote file exists

I don't have any idea why the issue occurs again although I confirmed the build works yesterday.
I reinstalled the newest release of wsl-vpnkit, but in vain.

Attached below is information about my environment. Could you check this?
wsl_command_output.log

Regards,

sakai135 · 2021-10-14T07:39:50Z

@tfl-takashima What do you get when running sudo iptables -t nat -S?

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -d 172.23.48.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.67.1:53
-A PREROUTING -d 172.23.48.1/32 -j DNAT --to-destination 192.168.67.2
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -d 172.23.48.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.67.1:53
-A OUTPUT -d 172.23.48.1/32 -j DNAT --to-destination 192.168.67.2
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -o eth1 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN

Also for docker run -it --rm busybox cat /etc/resolv.conf?

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateResolvConf = false
nameserver 172.23.48.1

Does start/stop/starting wsl-vpnkit or wsl.exe --shutdown fix the issue?

tfl-takashima · 2021-10-14T08:38:52Z

@sakai135 , thank you for quick response.

The issue fixed after I rebooted my laptop. I re-close this issue, but please let me ask if rebooting doesn't work in the future.
* Restarting wsl-vpnkit service or wsl distibution did not work for my issue.

P.S here is my output:

$ sudo iptables -t nat -S
[sudo] password for taka-t: 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -d 172.21.32.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.67.1:53
-A PREROUTING -d 172.21.32.1/32 -j DNAT --to-destination 192.168.67.2
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -d 172.21.32.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 192.168.67.1:53
-A OUTPUT -d 172.21.32.1/32 -j DNAT --to-destination 192.168.67.2
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -o eth1 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN

Thanks again,

tfl-takashima · 2021-10-15T14:34:08Z

Hi @sakai135,

I found that alpine linux container seems not be able to resolve hostname by its default server.
Could you check the issue is reproducible on your side?

Here is my test result.
* the issue occurs intermittently. Sometimes the wget / curl command done successfully.

# name resolution fails on alpine linux container.
$ docker run -it --rm  alpine:3.14 wget -S --spider www.google.com
wget: bad address 'www.google.com'
# but name resolution success if you directly set dns to real nameservers 
$ docker run -it --rm  --dns {set external dns server} alpine:3.14 wget -S --spider www.google.com
Connecting to www.google.com (172.217.161.196:80)
  HTTP/1.1 200 OK
  Date: Fri, 15 Oct 2021 14:25:46 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Server: gws
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: 1P_JAR=2021-10-15-14; expires=Sun, 14-Nov-2021 14:25:46 GMT; path=/; domain=.google.com; Secure
  Set-Cookie: NID=511=cMabYSLxbdUKnT6FUghphlu87CUbEF3950CvVao_YvHDjiMm3PmpyaedlZy2tj9T410UoeKgw5LQozNdpzPVBtQba7twVyiIdVmwuz8nkrYMFCghRRXW_du0oM0X36TmmpPBGKs7-rs2nNNXvv9Ona4ccK0TchjXy8O2V7n-Viw; expires=Sat, 16-Apr-2022 14:25:46 GMT; path=/; domain=.google.com; HttpOnly
  Accept-Ranges: none
  Vary: Accept-Encoding
  Connection: close
  Transfer-Encoding: chunked

remote file exists
# centos7 can access to external site without any problems 
$ docker run -it --rm centos:7 curl --head www.google.com
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Date: Fri, 15 Oct 2021 14:26:31 GMT
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
Expires: Fri, 15 Oct 2021 14:26:31 GMT
Cache-Control: private
Set-Cookie: 1P_JAR=2021-10-15-14; expires=Sun, 14-Nov-2021 14:26:31 GMT; path=/; domain=.google.com; Secure
Set-Cookie: NID=511=gN4mznRD5RNL3WLNkLj_1wOKr7EUItxcQMThfYfM_xqIMjVOG5sS0yuDTT469RpcjuqPXZnNDglesG2TD19RiBl6f-ernGZwqKfb8pAl7eL6Gw1XWVJsWTAWzk3l9X_vR6zUYYJz3m2Jid1-rpbU255t-X6oBTOOdIszMreBaWc; expires=Sat, 16-Apr-2022 14:26:31 GMT; path=/; domain=.google.com; HttpOnly

Regards,

sakai135 · 2021-10-16T03:08:19Z

I can't seem to replicate the issue you're having. Could you try #57 (build) and show the logs from the following?

wsl.exe -d wsl-vpnkit cat /var/log/wsl-vpnkit.log | grep -a -e ping -e check -e iptables

Since CentOS is ok, I am guessing that this is due to something between VPNKit and how Alpine/BusyBox/musl does DNS.

#57 contains a change that routes the DNS and other requests to the WSL2 gateway/host instead of VPNKit if your VPN allows it. This lets VPNKit hande the network traffic but lets WSL2/Hyper-V handle DNS.

If you see the line ping to Windows host successful in the logs, then that means your VPN allows it. If it says ping to Windows host failed, then the workaround might just be setting your own DNS server in /etc/resolv.conf on your WSL2 distro that's running dockerd.

tfl-takashima · 2021-10-18T00:25:25Z

Hi @sakai135 ,

I tried the #57 and here are the result:
The ping command works successfully, but DNS access fails for both container and wsl host.

$ wsl.exe -d wsl-vpnkit cat /var/log/wsl-vpnkit.log | grep -a -e ping -e check -e iptables
cleaning up iptables...
iptables cleanup done
ping to Windows host successful #  <========== ping SUCCESSFUL
check: ping success to Windows host (172.29.64.1) IPv4
check: ping success to Windows host (192.168.67.2) IPv4
check: ping success to VPNKit gateway (192.168.67.1) IPv4
check: nslookup fail for example.com A
ping: bad address 'example.com'
check: ping fail to external host (example.com) IPv4
check: nslookup fail for example.com AAAA
ping: bad address 'example.com'
check: ping fail as expected to external host (example.com) IPv6
cleaning up iptables...
iptables cleanup done
ping to Windows host successful
check: ping success to Windows host (172.29.64.1) IPv4
check: ping success to Windows host (192.168.67.2) IPv4
check: ping success to VPNKit gateway (192.168.67.1) IPv4
check: nslookup fail for example.com A
ping: bad address 'example.com'
check: ping fail to external host (example.com) IPv4
check: nslookup fail for example.com AAAA
ping: bad address 'example.com'
check: ping fail as expected to external host (example.com) IPv6
check: wget fail for https://example.com
$ docker run -it --rm  alpine:3.14 cat /etc/resolv.conf
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateResolvConf = false
nameserver 172.29.64.1
$ docker run -it --rm  alpine:3.14 wget www.google.com #  <========== Alpine's (Container) name resolution fails
wget: bad address 'www.google.com'
$ curl --head www.google.com  #  <========== WSL's name resolution also fails
curl: (6) Could not resolve host: www.google.com

The issue seems to be specific to my environment, so I'll do workaround by tweaking docker's daemon.json.
I'll report to you if I can detect any causes in the future.

Thanks for your cooperation!

treyturner · 2022-01-20T17:12:57Z

This is regularly happening to me as well; DNS in alpine containers breaks until I stop and restart vpnkit:

tturner@N412D350:~/dev/inserter-test/docker/app/deploy$ docker run -it alpine ping example.com
ping: bad address 'example.com'
tturner@N412D350:~/dev/inserter-test/docker/app/deploy$ wsl.exe -d wsl-vpnkit service wsl-vpnkit stop
tturner@N412D350:~/dev/inserter-test/docker/app/deploy$ wsl.exe -d wsl-vpnkit service wsl-vpnkit start
tturner@N412D350:~/dev/inserter-test/docker/app/deploy$ docker run -it alpine ping example.com
PING example.com (93.184.216.34): 56 data bytes
64 bytes from 93.184.216.34: seq=0 ttl=57 time=41.549 ms
64 bytes from 93.184.216.34: seq=1 ttl=57 time=41.865 ms
64 bytes from 93.184.216.34: seq=2 ttl=57 time=42.024 ms
^C
--- example.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 41.549/41.812/42.024 ms

tturner@N412D350:~/dev/inserter-test/docker/app/deploy$ wsl.exe -d wsl-vpnkit cat /var/log/wsl-vpnkit.log | grep -a -e ping -e check -e iptables
cleaning up iptables...
iptables cleanup done
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.17.32.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.17.32.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.17.32.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.28.192.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.28.192.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.28.192.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd
cleaning up iptables...
iptables cleanup done
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.28.192.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.28.192.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.28.192.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd
cleaning up iptables...
iptables cleanup done
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.28.192.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.28.192.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.28.192.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd
cleaning up iptables...
iptables cleanup done
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.28.192.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.28.192.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.28.192.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd
adding rules to iptables...
iptables done
check: ✔️ ping success to IPv4 WSL 2 gateway / Windows host (172.28.192.1)
check: ✔️ ping success to IPv4 VPNKit Windows host (192.168.67.2)
check: ✔️ ping success to IPv4 VPNKit gateway (192.168.67.1)
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 192.168.67.1
check: ✔️ nslookup success for example.com A using 172.28.192.1
check: ❌ nslookup fail for example.com A using 1.1.1.1
check: ✔️ ping success to IPv4 external host (example.com)
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 192.168.67.1
check: ❌ nslookup fail for example.com AAAA using 172.28.192.1
check: ❌ nslookup fail for example.com AAAA using 1.1.1.1
ping: bad address 'example.com'
check: ➖ ping fail to IPv6 external host (example.com)
check: ❌ wget fail for https://example.com
cleaning up iptables...
iptables cleanup done
stopping vpnkit-tap-vsockd

sakai135 · 2022-01-23T16:28:04Z

@treyturner Can you try setting your own DNS servers in /etc/resolv.conf of your Docker host and see if that fixes the issue?

sakai135 mentioned this issue Oct 12, 2021

use iptables prerouting to route other networks #48

Merged

tfl-takashima closed this as completed Oct 12, 2021

tfl-takashima reopened this Oct 14, 2021

tfl-takashima closed this as completed Oct 14, 2021

tfl-takashima reopened this Oct 15, 2021

tfl-takashima closed this as completed Oct 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker CE (Not Docker Desktop) containers can not connect to its nameserver. #47

Docker CE (Not Docker Desktop) containers can not connect to its nameserver. #47

tfl-takashima commented Oct 12, 2021 •

edited

Loading

sakai135 commented Oct 12, 2021

tfl-takashima commented Oct 12, 2021

tfl-takashima commented Oct 14, 2021

sakai135 commented Oct 14, 2021

tfl-takashima commented Oct 14, 2021 •

edited

Loading

tfl-takashima commented Oct 15, 2021 •

edited

Loading

sakai135 commented Oct 16, 2021

tfl-takashima commented Oct 18, 2021

treyturner commented Jan 20, 2022

sakai135 commented Jan 23, 2022

Docker CE (Not Docker Desktop) containers can not connect to its nameserver. #47

Docker CE (Not Docker Desktop) containers can not connect to its nameserver. #47

Comments

tfl-takashima commented Oct 12, 2021 • edited Loading

sakai135 commented Oct 12, 2021

tfl-takashima commented Oct 12, 2021

tfl-takashima commented Oct 14, 2021

sakai135 commented Oct 14, 2021

tfl-takashima commented Oct 14, 2021 • edited Loading

tfl-takashima commented Oct 15, 2021 • edited Loading

sakai135 commented Oct 16, 2021

tfl-takashima commented Oct 18, 2021

treyturner commented Jan 20, 2022

sakai135 commented Jan 23, 2022

tfl-takashima commented Oct 12, 2021 •

edited

Loading

tfl-takashima commented Oct 14, 2021 •

edited

Loading

tfl-takashima commented Oct 15, 2021 •

edited

Loading