Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports #10494

Closed
1 task done
halimsamy opened this issue Sep 19, 2023 · 169 comments
Closed
1 task done

WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports #10494

halimsamy opened this issue Sep 19, 2023 · 169 comments
Labels

Comments

@halimsamy
Copy link

halimsamy commented Sep 19, 2023

Windows Version

Microsoft Windows [Version 10.0.22631.2338]

WSL Version

2.0.0.0

Are you using WSL 1 or WSL 2?

  • WSL 2

Kernel Version

5.15.123.1-1

Distro Version

Ubuntu 20.04.6

Other Software

Docker version 24.0.6, build ed223bc

Repro Steps

  • Change networkingMode to mirrored
  • docker run -d -p 8080:80 nginx:alpine (example)
  • Go to localhost:8080
  • Cannot connect
  • Try from WSL curl http//localhost:8080 same issue

Expected Behavior

To forward the port and be able to connect to my containers

Actual Behavior

Doesn't forward the port, so I cannot connect to my containers.

Diagnostic Logs

No response

@driver1998
Copy link

driver1998 commented Sep 20, 2023

Interestingly, the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.

Machine A:

  • Change networkingMode to mirrored
  • docker run -d -p 8080:80 nginx:alpine
  • Go to localhost:8080
  • Cannot connect
  • Set Windows firewall rules to allow 8080 inbound
  • Go to localhost:8080
  • Cannot connect

Machine B:

  • Go to A:8080
  • works

@greenhat616
Copy link

greenhat616 commented Sep 20, 2023

Same issue


Maybe related issues:
docker/for-win#13686

@giovannicandido
Copy link

Same problem here

@halimsamy halimsamy changed the title networkingMode=mirrored makes Docker unable to forward ports WSL 2.0.0: networkingMode=mirrored makes Docker unable to forward ports Sep 20, 2023
@lengthofrope
Copy link

Yup, it's unfortunate but I have the same issue. I am on the release channel with windows version 10.0.22621.2359

@halimsamy halimsamy changed the title WSL 2.0.0: networkingMode=mirrored makes Docker unable to forward ports WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports Sep 20, 2023
@halimsamy halimsamy changed the title WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports WSL 2.0: networkingMode=mirrored makes Docker unable to forward ports Sep 20, 2023
@over-star
Copy link

Same problem here

@speller
Copy link

speller commented Sep 22, 2023

In my case, Docker ports are not exposed to the host Windows machine at all after the upgrade to 2.0 even if I set networkingMode=NAT which is the default value. I can't see opened ports in the tcpview.exe utility. Previously, I was able to see them opened by the wslhost.exe process (or similar name).

Switching to NAT helped me.

@Yukari316
Copy link

same here

@tuxiaobei-scu
Copy link

+1

2 similar comments
@bkfino
Copy link

bkfino commented Sep 24, 2023

+1

@ptr1120
Copy link

ptr1120 commented Sep 24, 2023

+1

@halimsamy
Copy link
Author

@benhillis Any ideas/updates here?

@DukeCrimson
Copy link

same here

@tumugin
Copy link

tumugin commented Sep 24, 2023

+1

@Aerglonus
Copy link

Aerglonus commented Sep 25, 2023

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

@halimsamy
Copy link
Author

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

I think that's a workaround that would work... but I am waiting for someone from the WSL team to answer us here, did they get to know that is the issue, and if they have any plans to fix this?

@lengthofrope
Copy link

Same issue here, the only way I found to make it "work" was adding ignoredPorts=8080 to the wslconfig, but if container has something like 4800:8080 need to add both ports to ignoredPorts=8080,4800 still doing this some containers might not work at all, also if cloudflare tunnel is running on WSL this will break to when mirrored and theres no way to know which ports is using to add it to the ignoredPorts

That seems to work. Nice find.

@zcobol
Copy link

zcobol commented Sep 25, 2023

ignorePorts is misleading. What it does is making in this case port 8080 available to request inside the container only, but is not forwarded to Windows side. Look for more details at https://learn.microsoft.com/en-us/windows/wsl/wsl-config

If you run get-nettcpConnection -LocalPort 8080 there's nothing listening on that port.

After starting the web server with docker run -d -p 8080:80 nginx:alpine requests inside the WSL container works:

zcobol@debian:~$ curl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:41:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes

Requests from Windows side timeout with:

curl: (28) Failed to connect to localhost port 8080 after 21276 ms: Couldn't connect to server

From another server it work, as mentioned by @driver1998 :

tux@raspi:~$ curl -I 192.168.1.105:8080
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Mon, 25 Sep 2023 16:43:45 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 19:24:07 GMT
Connection: keep-alive
ETag: "64dbd0d7-267"
Accept-Ranges: bytes

@mew1033
Copy link

mew1033 commented Sep 25, 2023

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125
I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

@halimsamy
Copy link
Author

Just ran into this as well. AWS sam build --use-container doesn't work. Gives the same error

 Ports are not available: exposing port TCP 127.0.0.1:5232 -> 0.0.0.0:0: listen tcp 127.0.0.1:5232: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

(The port number changes randomly every time)

Based on this: https://github.com/aws/aws-sam-cli/blob/c5b9b1e399a1e5c938ef72934a14ede934e17bac/samcli/local/docker/container.py#L124-L125 I added every port from 5000-9000 to the ignoredPorts list. It's ugly, but for now it works.

It seems like it's generally a problem with WSL itself. I am looking for a fix soon, since the new network mode is so much useful but it's not useable (usefully) in the current state.

@imacarthur741
Copy link

I ma having the same issue with apache. No changes other than added --experimental but now nothing works.

(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
[no listening sockets available, shutting down](apache2.service: Control process exited, code=exited, status=1/FAILURE)

root@ACER-Nitro:/usr/sbin# lsof -nP -iTCP -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 308 systemd-resolve 14u IPv4 22507 0t0 TCP 127.0.0.53:53 (LISTEN)
mysqld 385 mysql 21u IPv4 32886 0t0 TCP 127.0.0.1:33060 (LISTEN)
mysqld 385 mysql 23u IPv4 27396 0t0 TCP 127.0.0.1:3306 (LISTEN)
miniserv. 1867 root 5u IPv4 52863 0t0 TCP *:10000 (LISTEN)

@zohaibhassan156
Copy link

Same problem

@zohaibhassan156
Copy link

ignoredPorts=8025,1025 work for me too. I had to use mailpit in docker

docker run -d \
--name=mailpit \
--restart unless-stopped \
-p 8025:8025 \
-p 1025:1025 \
axllent/mailpit

@shigenobuokamoto
Copy link

there seem to be two issues why Docker containers cannot connect from Windows.

  1. ignorePorts works
    Docker Desktop, probably

  2. the port forwarding does work from another machine on the same network as host. Just not on the host machine itself.
    docker-ce that installed on Linux

temporary measures for 2 ....
disable iptables and use docker-proxy

/etc/docker/daemon.json

{
    "iptables": false
}

when using mirrored, the behavior seems to be different from the previous localhostforwarding.

use docker-proxy(listen on Linux)

windows  --->  linux(WSL)
  localhostforwarding (until)
ok    127.0.0.1   --->     127.0.0.1(lo)
ok    127.0.0.1   <---     127.0.0.1(lo)

  netowrkingMode=mirrored
ok    127.0.0.1   --->     127.0.0.1(loopback0)
ok    127.0.0.1   <---     127.0.0.1(loopback0)

interface is different, but the behavior remains the same.

use iptables(listeon on container)

windows  --->  linux(WSL) ---> Docker container
  localhostforwarding (until)
ok    127.0.0.1   ---> 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  --->  172.x.x.x(eth0)
ok    127.0.0.1   <--- 127.0.0.1(lo)    /   172.x.x.1(br-xxx)  <---  172.x.x.x(eth0)

  netowrkingMode=mirrored
ok    127.0.0.1   ---> 127.0.0.1(loopback0)/127.0.0.1(br-xxxx) --->  172.x.x.x(eth0)
                                          ~~~~~~~~~
ng                                                           <- - -  172.x.x.x(eth0)

via localhostforwarding(until), source address(Windows) was the docker network gateway (=pointing to linux).

via mirrored, source address is 127.0.0.1.
for this reason that packet is returned to 127.0.0.1(localhost on container) and does not reach Windows.
in the case of access from another node(PC), there is no problem because source is his address.

@shigenobuokamoto
Copy link

@keith-horton
(https://github.com/microsoft/WSL/releases/tag/2.3.11)[WSL 2.3.11]

Update mirrored mode loopback routing to support connections between Windows host and Linux Docker/Podman containers (solves #11136,#11468,#11758,#10926)

this improvement gives us more options.

in the daemon.json

"userland-proxy": false

this now works as well.

but

moby/moby#48075

this change is made, 'userland-proxy: false' may no longer work.
i think it needs a little adjustment.

@shigenobuokamoto
Copy link

this may seem unnecessary since moby takes care of it, but this is a new improvement plan.

WSL 2.3.11 brings several improvements to communication with Windows host.

therefore, you can now use the following rules to improve communication from Windows host to Docker containers:

$ nft add rule ip nat WSLPOSTROUTING iif "loopback0" ip saddr 127.0.0.0/8 ip daddr != 127.0.0.0/8 counter masquerade
$ sysctl -w net.ipv4.conf.all.route_localnet=1

@codeart1st
Copy link

@shigenobuokamoto with WSL pre-release 2.3.11 would you still recommend your systemd script workaround?

https://gist.github.com/shigenobuokamoto/b565d468541fc8be7d7d76a0434496a0

@felipecrs
Copy link

@shigenobuokamoto with WSL pre-release 2.3.11 would you still recommend your systemd script workaround?

https://gist.github.com/shigenobuokamoto/b565d468541fc8be7d7d76a0434496a0

As an user, I can it still works fine.

@shigenobuokamoto
Copy link

@codeart1st
work on this issue is ongoing in moby (=Docker Engine), so until that is released you can use my script to mitigate the issue.
essentially, the issue is that network within WSL behaves differently from normal Linux, so i think it would be best to deal with it on the WSL side.

WSL 2.3.11 includes some improvements to mirrored networking, so i am looking into how to use them to improve the experience. this is the prototype new network-mirrored.service. please try this too.

https://gist.github.com/shigenobuokamoto/540c5f09a03eb07149501e99a6c8d82b

  • taking advantage of the fact that connect from Docker container network to Windows host is now possible, i wrote a more natural nftables.
  • added support for communication between Windows and Linux using 127.0.0.0/8, so i worte the missing route.

@felipecrs
Copy link

the issue is that network within WSL behaves differently from normal Linux, so i think it would be best to deal with it on the WSL side.

Also, this would mean that whatever application that relies on the same feature as docker would have the same issue. It could be an old application that doesn't receive more support for example. Or some application that refuses to implement a fix just for WSL.

So, yeah, I totally agree. The ideal would have been the fix to happen on WSL side.

@Parsifa1
Copy link

Parsifa1 commented Jul 21, 2024

When will this fix be built into wsl? Now we can only use service script to make it work, I don't think this is a long term solution.

Besides, I use nixos-wsl, this is my example module that can be used as a reference for nixos user, from @shigenobuokamoto

{pkgs, ...}: {
    systemd.services.network-mirrored = {
      description = "network-mirrored";
      enable = true;
      wants = ["network-pre.target"];
      wantedBy = ["multi-user.target"];
      before = ["network-pre.target" "shutdown.target"];
      serviceConfig = {
        User = "root";
        ExecStart = [
          ''
            /bin/sh -ec '\
            [ -x /usr/bin/wslinfo ] && [ "$(/usr/bin/wslinfo --networking-mode)" = "mirrored" ] || exit 0;\
            echo "\
            add chain   ip nat WSLPREROUTING { type nat hook prerouting priority dstnat - 1; policy accept; };\
            insert rule ip nat WSLPREROUTING iif loopback0  ip daddr 127.0.0.1 counter dnat to 127.0.0.1 comment mirrored;\
            "|${pkgs.nftables}/bin/nft -f -\
            '
          ''
        ];

        ExecStop = [
          ''
            /bin/sh -ec '\
              [ -x /usr/bin/wslinfo ] && [ "$(/usr/bin/wslinfo --networking-mode)" = "mirrored" ] || exit 0;\
              for chain in "ip nat WSLPREROUTING";\
              do\
                handle=$(${pkgs.nftables}/bin/nft -a list chain $chain | sed -En "s/^.*comment \\"mirrored\\" # handle ([0-9]+)$/\\1/p");\
                for n in $handle; do echo "delete rule $chain handle $n"; done;\
              done|${pkgs.nftables}/bin/nft -f -\
            '
          ''
        ];
        RemainAfterExit = "yes";
      };
    };
}

@zaaack
Copy link

zaaack commented Aug 9, 2024

here is a really dirty solution, I just wrote a nodejs tcp reverse proxy server to map the docker port to another port, so we can using another port because its created by a non-docker process.

import net from 'net'
let map = {
  15432: 5432,
  13000: 3000,
}
for (const from in map) {
  const to = map[from]
  net
    .createServer((server) => {
      const client = net.createConnection(to)
      server.pipe(client)
      client.pipe(server)
    })
    .listen(from)
    .on('listening', () => {
      console.log(`tcp-proxy: ${from} -> ${to}`)
    })
    .on('error', (err) => {
      console.error(err)
    })
}

@felipecrs
Copy link

@zaaack, if you are looking for a temporary solution, this one is the best you'll find: #10494 (comment)

@zaaack
Copy link

zaaack commented Aug 9, 2024

@zaaack, if you are looking for a temporary solution, this one is the best you'll find: #10494 (comment)

thanks, but my wsl is older and doesnt set systemd

@dannyhpy
Copy link

dannyhpy commented Aug 10, 2024

thanks, but my wsl is older and doesnt set systemd

@zaaack, you can use iptables -t nat -I DOCKER -i loopback0 -d 127.0.0.0/8 -j RETURN¹ or any of the variants² that have been mentioned above.

@jweaston
Copy link

jweaston commented Aug 16, 2024

Has anyone had success configuring WSL to allow connections from containers in WSL to the windows host?
I made some attempts on my own to forward used ports, 5966 in this case, from WSL to the windows host through the nft but was unsuccesfull.
Maybe some one can tell me what I did wrong.

sudo nft add chain ip nat prerouting { type nat hook prerouting priority 0 \; }
sudo nft add chain ip nat postrouting { type nat hook postrouting priority 100 \; }
sudo nft add rule ip nat prerouting tcp dport 5966 counter dnat to 127.0.0.1:5966
sudo nft add rule ip nat postrouting ip saddr 172.0.0.0/8 counter masquerade

@shigenobuokamoto
Copy link

@jweaston
you can access from Docker container in WSL to Windows host address, WSL 2.3.11 and later.

.wslconfig

[experimental]
hostAddressLoopback=true
Windows host                  WSL Linux                         Docker Container
127.0.0.1    --- loopback --- 127.0.0.1   --- not connected --- 127.0.0.1(same address but unrelated)
192.168.1.10 --- loopback --- 192.168.1.10
                                  | routed
     (unknown network)        172.17.0.1  ----- connected ----- 172.17.0.2

docker container (in WSL) can connect to 192.168.1.10:5966.
if you try connect to 127.0.0.1:5966, the packet is sent to container's 127.0.0.1. so cannot connect to windows host.

@CatalinFetoiu
Copy link
Collaborator

hello. Docker completed the PR that fixes this issue - moby/moby#48514
This was included in the following Docker release - https://github.com/moby/moby/releases/tag/v27.3.0

Please try Docker version 27.3.0

@felipecrs
Copy link

It works in my test. :)

@unclemusclez
Copy link

My favorite flavor of Linux is Ubuntu via Docker via Ubuntu via WSL via Windows 11.

@BolteDev
Copy link

BolteDev commented Sep 22, 2024

Just to add to this...

I recently set up Ubuntu 24.04.1 with Docker and it works fine in Mirrored mode (Docker 27.3.1)
But on my Alpine install (Docker 26.1.5) it refuses to work unless I set it back to NAT.

Was driving me crazy

EDIT:
Does anyone have any advice on manually installing latest version in Alpine without using Dockers setup script that tries to force mt to use Docker Desktop?

@unclemusclez
Copy link

Just to add to this...

I recently set up Ubuntu 24.04.1 with Docker and it works fine in Mirrored mode (Docker 27.3.1) But on my Alpine install (Docker 26.1.5) it refuses to work unless I set it back to NAT.

Was driving me crazy

EDIT: Does anyone have any advice on manually installing latest version in Alpine without using Dockers setup script that tries to force mt to use Docker Desktop?

@BolteDev can i see all of your relevant settings because i still cant get it to work. I am trying to run a simple gradio app inside of a docker instance. this used to work fine and since i've started trouble shooting everything i have no idea what are the proper settings.

@BolteDev
Copy link

Just to add to this...
I recently set up Ubuntu 24.04.1 with Docker and it works fine in Mirrored mode (Docker 27.3.1) But on my Alpine install (Docker 26.1.5) it refuses to work unless I set it back to NAT.
Was driving me crazy
EDIT: Does anyone have any advice on manually installing latest version in Alpine without using Dockers setup script that tries to force mt to use Docker Desktop?

@BolteDev can i see all of your relevant settings because i still cant get it to work. I am trying to run a simple gradio app inside of a docker instance. this used to work fine and since i've started trouble shooting everything i have no idea what are the proper settings.


I haven't done anything different than following normal installations and setting up docker, other than my wslconfig having networkingMode=mirrored.

I was setting up an Alpine install and nginx just wasn't working until I disabled it and went back to regular NAT along with localhostForwarding=true, I thought the docker just wasn't working until I did a curl against localhost from within the instance and realised it was working, which lead me here.

I'm still pretty new to Linux side of things, but I would say if you are lost with it, start from scratch. I would say try and get something simple like Nginx working then at least you know the networking side works and it will then be a docker image issue or not.

But basically all I did was install openrc (I'm using the 3mb rootfs Alpine) do the basic setup to start it in /etc/init-wsl:

# OpenRC
mount --make-rshared /
mkdir -p /run/openrc
touch /run/openrc/softlevel
sysctl net.ipv4.ip_unprivileged_port_start=80 > /dev/null 2>&1

openrc
rc-service networkmanager start
rc-service docker start

Then ran the standard nginx image as a test on 80:80

@keith-horton
Copy link
Member

If you're seeing issues, please be sure you have the latest Docker + latest WSL.
Some applications may resolve the name of the local device to an assigned address - instead of using loopback. In those cases you'll want to set [experimental] hostAddressLoopback=true.

https://learn.microsoft.com/en-us/windows/wsl/wsl-config

hostAddressLoopback (bool - default == false)
Only applicable when wsl2.networkingMode is set to mirrored. When set to True, will allow the Container to connect to the Host, or the Host to connect to the Container, by an IP address that's assigned to the Host. The 127.0.0.1 loopback address can always be used,this option allows for all additionally assigned local IP addresses to be used as well. Only IPv4 addresses assigned to the host are supported.

@unclemusclez
Copy link

If you're seeing issues, please be sure you have the latest Docker + latest WSL. Some applications may resolve the name of the local device to an assigned address - instead of using loopback. In those cases you'll want to set [experimental] hostAddressLoopback=true.

https://learn.microsoft.com/en-us/windows/wsl/wsl-config

hostAddressLoopback (bool - default == false) Only applicable when wsl2.networkingMode is set to mirrored. When set to True, will allow the Container to connect to the Host, or the Host to connect to the Container, by an IP address that's assigned to the Host. The 127.0.0.1 loopback address can always be used,this option allows for all additionally assigned local IP addresses to be used as well. Only IPv4 addresses assigned to the host are supported.

@keith-horton so i want them both to true? i run other apps withing WSL that work fine, but when i try to host the docker containers from the run command inside of WSL it does not connect to a forwarded port.

@keith-horton
Copy link
Member

Right: some apps may want to connect to an assigned address instead of loopback. In those cases, it might help those solutions end-to-end to also enable hostAddressLoopback

(hostAddressLoopback is only applicable to Mirrored mode)

@unclemusclez
Copy link

Right: some apps may want to connect to an assigned address instead of loopback. In those cases, it might help those solutions end-to-end to also enable hostAddressLoopback

(hostAddressLoopback is only applicable to Mirrored mode)

@keith-horton brother, you just changed my life. thank you

@codeart1st
Copy link

docker --version
Docker version 27.3.1, build ce12230

wsl.exe --version
WSL version: 2.3.14.0
Kernel version: 6.6.36.3-1
WSLg version: 1.0.64
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.22631.4169

Disabled and stopped the systemd service helper for https://gist.github.com/shigenobuokamoto/540c5f09a03eb07149501e99a6c8d82b.

Propably most things work, but for no real reason I can't connect with my Oracle DB Container anymore. Reenabled the systemd service helper and my Oracle DB Container works again.

@CatalinFetoiu
Copy link
Collaborator

closing since the issue is resolved
if you encounter the issue after using Docker version 27.3.0 or later, please open a separate issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests