Cannot map port 53 from container to host; conflicts with dnsmasq

[james-crowley]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

When using docker-compose and podman, podman fails to bring up containers trying to port map ports below 60. Additionally, when trying to map port 53 on the host, it seems to conflict with dnsmasq process podman spawns.

Steps to reproduce the issue:

Parsing Error

1. Install podman 3.0 as root to utilize docker-compose features

2. Make sure to disable any dns(port 53) service running on OS

3. Using the docker-compose.yml file below issue: docker-compose up

Port 53 Conflict

1. Install podman 3.0 as root to utilize docker-compose features

2. Make sure to disable any dns(port 53) service running on OS

3. Edit the docker-compose.yml file and change - 53:53 to - 53:XXXX, where XXXX is anything above 59.
   Example: - 53:60

4. Then issue the following: docker-compose up

Describe the results you received:

Using the unmodified docker-compose.yml file below will generate the parsing error:

root@vm-307:/home/crowley# docker-compose up
Creating network "crowley_default" with the default driver
Creating crowley_admin_1 ...
Creating crowley_pdns_1  ... error
Creating crowley_admin_1 ... done
ERROR: for crowley_pdns_1  Cannot create container for service pdns: make cli opts(): strconv.Atoi: parsing "": invalid syntax

From my testing if I change the port mapping, - 53:53 to be anything above 59 for the container port, it passes the parsing error.

Changing the port mapping to - 53:60, allows the docker-compose up to continue but fail with this error message:

root@vm-307:/home/crowley# docker-compose up
Creating network "crowley_default" with the default driver
Creating crowley_admin_1 ...
Creating crowley_pdns_1  ... error
Creating crowley_admin_1 ... done
ERROR: for crowley_pdns_1  error preparing container ac8f5caddef9e28d43fd2f8b41d0c96845765c623b1f7fe0fef3b6692efa5910 for attach: cannot listen on the TCP port: listen tcp4 :53: bind: address already in use

ERROR: for pdns  error preparing container ac8f5caddef9e28d43fd2f8b41d0c96845765c623b1f7fe0fef3b6692efa5910 for attach: cannot listen on the TCP port: listen tcp4 :53: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Just to make sure I am not crazy, I bring down the containers, docker-compose down. Then check my ports using sudo lsof -i -P -n. Which results in:

root@vm-307:/home/crowley# sudo lsof -i -P -n
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    630    root    3u  IPv4  32734      0t0  TCP *:22 (LISTEN)
sshd    630    root    4u  IPv6  32736      0t0  TCP *:22 (LISTEN)
sshd    668    root    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)
sshd    695 crowley    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)

Please note X.X.X.X is just me censoring my IPs. As you can see I do not have any services listen on port 53.

Next I issue docker-compose up again. I see the same port conflict issue. Then issue sudo lsof -i -P -n to check my services before bringing down the containers.

root@vm-307:/home/crowley# sudo lsof -i -P -n
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd      630    root    3u  IPv4  32734      0t0  TCP *:22 (LISTEN)
sshd      630    root    4u  IPv6  32736      0t0  TCP *:22 (LISTEN)
sshd      668    root    4u  IPv4  32763      0t0  TCP X.X.X.X->X.X.X.X:55832 (ESTABLISHED)
sshd      695 crowley    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)
dnsmasq 16060    root    4u  IPv4 112910      0t0  UDP 10.89.0.1:53
dnsmasq 16060    root    5u  IPv4 112911      0t0  TCP 10.89.0.1:53 (LISTEN)
dnsmasq 16060    root   10u  IPv6 116160      0t0  UDP [fe80::9cc6:14ff:fe16:3953]:53
dnsmasq 16060    root   11u  IPv6 116161      0t0  TCP [fe80::9cc6:14ff:fe16:3953]:53 (LISTEN)
conmon  16062    root    5u  IPv4 111869      0t0  TCP *:9191 (LISTEN)

As you can see podman has spawned a dnsmasq process. I think this is to allow DNS between the containers, but seems to conflict if you want to run/port map port 53.

Describe the results you expected:

I expect not to hit that parsing error. I am not sure why podman/docker-compose is hitting that error. When running that exact same docker-compose.yml via docker I have no issues.

I also expect not to hit port 53 conflicts. I am not sure how podman is handling DNS between the containers but the implementation limits users ability to hosts different services.

Additional information you deem important (e.g. issue happens only occasionally):

N/A

Output of podman version:

podman version 3.0.0

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: vm-307
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-28-generic
  linkmode: dynamic
  memFree: 15873085440
  memTotal: 16762957824
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17.6-58ef-dirty
      commit: fd582c529489c0738e7039cbc036781d1d039014
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 1023406080
  swapTotal: 1023406080
  uptime: 1h 11m 7.15s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 100:3.0.0-4 amd64 [installed]
podman/unknown 100:3.0.0-4 arm64
podman/unknown 100:3.0.0-4 armhf
podman/unknown 100:3.0.0-4 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on amd64 hardware. The server is a VM inside of VMware. Also running on Ubuntu 20.04.

docker-compose.yml

version: "3"
services:
  pdns:
    image:  powerdns/pdns-auth-master:latest
    ports:
      - 53:53
      - 8081:8081
  admin:
    image: ngoduykhanh/powerdns-admin:latest
    ports:
      - 9191:80

[mheon]

To confirm, are you running Podman rootless or as root?

[james-crowley]

@mheon As root. As far as I know to use the docker-compose features you need to be running as root.

More information here: https://www.redhat.com/sysadmin/compose-kubernetes-podman

[baude]

@james-crowley can you provide the compose file in question or a simplified one?

[james-crowley]

@baude It's in the issue. I simplified it down already. It is at the bottom.

version: "3"
services:
 pdns:
   image:  powerdns/pdns-auth-master:latest
   ports:
     - 53:53
     - 8081:8081
 admin:
   image: ngoduykhanh/powerdns-admin:latest
   ports:
     - 9191:80

[baude]

ok, i can replicate this, i will need to dive in deeper. thanks for the issue and ill update here as soon as I can

[james-crowley]

@baude Let me know if you need any help. Happy to lend a hand. I got access to s390x and ppc64le, so I you want me to verify the bug exists on the other architectures let me know.

[baude]

the socat logs from this operation show this:

{"ExposedPorts": {"3233/tcp": {}, "8081/tcp": {}}, "Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Image": "powerdns/pdns-auth-master:latest", "Volumes": {}, "NetworkDisabled": false, "HostConfig": {"NetworkMode": "9444_default", "VolumesFrom": [], "Binds": [], "PortBindings": {"3233/tcp": [{"HostIp": "", "HostPort": ""}], "8081/tcp": [{"HostIp": "", "HostPort": "8081"}]}, "Links": [], "LogConfig": {"Type": "", "Config": {}}}, "NetworkingConfig": {"EndpointsConfig": {"9444_default": {"Aliases": ["pdns"], "IPAMConfig": {}}}}, "Labels": {"com.docker.compose.project": "9444", "com.docker.compose.service": "pdns", "com.docker.compose.oneoff": "False", "com.docker.compose.project.working_dir": "/home/baude/tmp/9444", "com.docker.compose.project.config_files": "docker-compose.yml", "com.docker.compose.container-number": "1", "com.docker.compose.version": "1.27.4", "com.docker.compose.config-hash": "cfced1b43ab39519b4d3b19b56a02dcc9db29dc0344c280d7857854452a2aaed"}}< 2021/02/19 15:00:53.366257  length=277 from=0 to=276

which is odd ... so it seems that podman is doing exactly what it's been told. map 3233 to a random port. so now, we should focus on why

[baude]

the collision on port 53 is due to dnsmasq running on the network created by compose. what happens if you temporarily rename the cni plugin called dnsname? on my system that is /usr/libexec/cni/dnsname

[james-crowley]

@baude Same path on my system. Renaming dnsname to dnsname_tmp cause the docker-compose up to run without the port conflict.

The problem comes now you are not able to use the internal DNS to resolve container names. The docker-compose.yml file was simplified. In my real file, I use two database containers in addition to the existing containers. The containers need to resolve the DNS name of the containers to connect to the database.

Thus I end up with the errors:

admin_1     | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "admin-db" to address: Name does not resolve
admin_1     |
admin_1     | (Background on this error at: http://sqlalche.me/e/13/e3q8)

Where admin-db is one of the database containers.

It seems like podman is implementing a different solution to DNS resolution between the containers? As this conflict does not happen with docker.

Is there away to tune/configure the cni plugin to not launch the service on the same network compose creates?

[baude]

yeah there is a way but not through compose. and i assume you want container to container name resolution. i have some ideas on this to discuss with the team and see if we can come up with a solution.

[james-crowley]

@baude Yeah, having container to container resolution what I am after. Having the docker-compose with podman behave the same or as close as possible to docker-compose with docker would be great.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[james-crowley]

@baude Any updates or testing help needed?

[FarisZR]

can replicate with pi-hole in docker-compose
fedora 33 server
podman v3.0.1
port 53 already in use by dnsmasq
this unfortunately a deal breaker, i can't use podman until this is fixed

[james-crowley]

@rhatdan @baude Any updates on this issue? Happy to test with any patches you might have.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

we should deal with this issue of dns ports in podman 4.0

[mheon]

Probably need to have a discussion about how...

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

this will be fixed in 4.0 with aardvark.

[Luap99]

I am not sure if this will be fixed for 4.0, @mheon right?

[baude]

if not, we should move this issue there

[openshift-ci]

@james-crowley: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to this:

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

When using docker-compose and podman, podman fails to bring up containers trying to port map ports below 60. Additionally, when trying to map port 53 on the host, it seems to conflict with dnsmasq process podman spawns.

Steps to reproduce the issue:

Parsing Error

1. Install podman 3.0 as root to utilize docker-compose features

2. Make sure to disable any dns(port 53) service running on OS

3. Using the docker-compose.yml file below issue: docker-compose up

Port 53 Conflict

1. Install podman 3.0 as root to utilize docker-compose features

2. Make sure to disable any dns(port 53) service running on OS

3. Edit the docker-compose.yml file and change - 53:53 to - 53:XXXX, where XXXX is anything above 59.
   Example: - 53:60

4. Then issue the following: docker-compose up

Describe the results you received:

Using the unmodified docker-compose.yml file below will generate the parsing error:

root@vm-307:/home/crowley# docker-compose up
Creating network "crowley_default" with the default driver
Creating crowley_admin_1 ...
Creating crowley_pdns_1  ... error
Creating crowley_admin_1 ... done
ERROR: for crowley_pdns_1  Cannot create container for service pdns: make cli opts(): strconv.Atoi: parsing "": invalid syntax

From my testing if I change the port mapping, - 53:53 to be anything above 59 for the container port, it passes the parsing error.

Changing the port mapping to - 53:60, allows the docker-compose up to continue but fail with this error message:

root@vm-307:/home/crowley# docker-compose up
Creating network "crowley_default" with the default driver
Creating crowley_admin_1 ...
Creating crowley_pdns_1  ... error
Creating crowley_admin_1 ... done
ERROR: for crowley_pdns_1  error preparing container ac8f5caddef9e28d43fd2f8b41d0c96845765c623b1f7fe0fef3b6692efa5910 for attach: cannot listen on the TCP port: listen tcp4 :53: bind: address already in use

ERROR: for pdns  error preparing container ac8f5caddef9e28d43fd2f8b41d0c96845765c623b1f7fe0fef3b6692efa5910 for attach: cannot listen on the TCP port: listen tcp4 :53: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Just to make sure I am not crazy, I bring down the containers, docker-compose down. Then check my ports using sudo lsof -i -P -n. Which results in:

root@vm-307:/home/crowley# sudo lsof -i -P -n
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    630    root    3u  IPv4  32734      0t0  TCP *:22 (LISTEN)
sshd    630    root    4u  IPv6  32736      0t0  TCP *:22 (LISTEN)
sshd    668    root    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)
sshd    695 crowley    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)

Please note X.X.X.X is just me censoring my IPs. As you can see I do not have any services listen on port 53.

Next I issue docker-compose up again. I see the same port conflict issue. Then issue sudo lsof -i -P -n to check my services before bringing down the containers.

root@vm-307:/home/crowley# sudo lsof -i -P -n
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd      630    root    3u  IPv4  32734      0t0  TCP *:22 (LISTEN)
sshd      630    root    4u  IPv6  32736      0t0  TCP *:22 (LISTEN)
sshd      668    root    4u  IPv4  32763      0t0  TCP X.X.X.X->X.X.X.X:55832 (ESTABLISHED)
sshd      695 crowley    4u  IPv4  32763      0t0  TCP X.X.X.X:22->X.X.X.X:55832 (ESTABLISHED)
dnsmasq 16060    root    4u  IPv4 112910      0t0  UDP 10.89.0.1:53
dnsmasq 16060    root    5u  IPv4 112911      0t0  TCP 10.89.0.1:53 (LISTEN)
dnsmasq 16060    root   10u  IPv6 116160      0t0  UDP [fe80::9cc6:14ff:fe16:3953]:53
dnsmasq 16060    root   11u  IPv6 116161      0t0  TCP [fe80::9cc6:14ff:fe16:3953]:53 (LISTEN)
conmon  16062    root    5u  IPv4 111869      0t0  TCP *:9191 (LISTEN)

As you can see podman has spawned a dnsmasq process. I think this is to allow DNS between the containers, but seems to conflict if you want to run/port map port 53.

Describe the results you expected:

I expect not to hit that parsing error. I am not sure why podman/docker-compose is hitting that error. When running that exact same docker-compose.yml via docker I have no issues.

I also expect not to hit port 53 conflicts. I am not sure how podman is handling DNS between the containers but the implementation limits users ability to hosts different services.

Additional information you deem important (e.g. issue happens only occasionally):

N/A

Output of podman version:

podman version 3.0.0

Output of podman info --debug:

host:
 arch: amd64
 buildahVersion: 1.19.2
 cgroupManager: systemd
 cgroupVersion: v1
 conmon:
   package: 'conmon: /usr/libexec/podman/conmon'
   path: /usr/libexec/podman/conmon
   version: 'conmon version 2.0.26, commit: '
 cpus: 8
 distribution:
   distribution: ubuntu
   version: "20.04"
 eventLogger: journald
 hostname: vm-307
 idMappings:
   gidmap: null
   uidmap: null
 kernel: 5.4.0-28-generic
 linkmode: dynamic
 memFree: 15873085440
 memTotal: 16762957824
 ociRuntime:
   name: crun
   package: 'crun: /usr/bin/crun'
   path: /usr/bin/crun
   version: |-
     crun version 0.17.6-58ef-dirty
     commit: fd582c529489c0738e7039cbc036781d1d039014
     spec: 1.0.0
     +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
 os: linux
 remoteSocket:
   path: /run/podman/podman.sock
 security:
   apparmorEnabled: true
   capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
   rootless: false
   seccompEnabled: true
   selinuxEnabled: false
 slirp4netns:
   executable: ""
   package: ""
   version: ""
 swapFree: 1023406080
 swapTotal: 1023406080
 uptime: 1h 11m 7.15s (Approximately 0.04 days)
registries:
 search:
 - docker.io
 - quay.io
store:
 configFile: /etc/containers/storage.conf
 containerStore:
   number: 4
   paused: 0
   running: 0
   stopped: 4
 graphDriverName: overlay
 graphOptions:
   overlay.mountopt: nodev,metacopy=on
 graphRoot: /var/lib/containers/storage
 graphStatus:
   Backing Filesystem: extfs
   Native Overlay Diff: "false"
   Supports d_type: "true"
   Using metacopy: "true"
 imageStore:
   number: 4
 runRoot: /run/containers/storage
 volumePath: /var/lib/containers/storage/volumes
version:
 APIVersion: 3.0.0
 Built: 0
 BuiltTime: Wed Dec 31 19:00:00 1969
 GitCommit: ""
 GoVersion: go1.15.2
 OsArch: linux/amd64
 Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 100:3.0.0-4 amd64 [installed]
podman/unknown 100:3.0.0-4 arm64
podman/unknown 100:3.0.0-4 armhf
podman/unknown 100:3.0.0-4 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on amd64 hardware. The server is a VM inside of VMware. Also running on Ubuntu 20.04.

docker-compose.yml

version: "3"
services:
 pdns:
   image:  powerdns/pdns-auth-master:latest
   ports:
     - 53:53
     - 8081:8081
 admin:
   image: ngoduykhanh/powerdns-admin:latest
   ports:
     - 9191:80

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mheon]

This will not be 4.0 unless we get very ambitious.

[mheon]

Do we have a card for the free-up-53 work in Aardvark?

[baude]

no

[d-513]

deal breaker for me.

[martinetd]

What's wrong with adding an iptables rule like the one suggested by #13 (comment) -- assuming we also filter by source IP or interface ?

Running manually with the following

# /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 1153 run
# iptables -t nat -A PREROUTING -s 10.88.0.0/16 -d 10.88.0.1/32 -p udp --dport 53 -j DNAT --to-destination 10.88.0.1:1153

seems to work as expected, so it doesn't look too bad?

(FWIW I'm now struggling after containers/podman#14412 as I added a dnsmasq so updating nameservers by changing network would work, fell into this same problem, made dnsmasq run with bind-interfaces so it wouldn't conflict, but now running into more problems with that dnsmasq not starting up reliably when adding more interfaces into the mix so I started looking into this just now instead -- feels much less hacky to free up the 53 port here instead of pouncing on dnsmasq some more)

@f1yn nameserver 127.0.0.1#153 in resolv.conf didn't work on musl at least (didn't try glibc resolver), so that probably shouldn't be relied on.

[d-513]

Solution that worked for me was bind the DNS from the container to the IP of the machine directly. Then it doesn’t conflict with aavark-dns. But in resolv.conf, you need to use machine’s ip, not 127.0.0.1

[martinetd]

Yes, that's what bind-interfaces in dnsmasq does, but it's far from ideal when some interfaces show up later e.g. bridge created by hostapd, which can also disappear and reappear, and dnsmasq's normal way of dealing with that is... not using bind-interfaces, so we're back here :)

I've just had a look at the netavark code that spawns aardvark-dns (this issue probably ought to move there?) and there's a bit of plumbing to do since we don't currently have in PREROUTING, but it doesn't look too bad. I don't have time to do it immediately but if it's not done by mid-July I'll probably be able to justify spending some time on it... Which is by no mean an invitation to wait for me, but at least I'll try!

[f1yn]

@martinetd In order to access my my DNS server consistently over both my LAN, but also remotely via WireGuard - I gave up having the DNS port remapped and instead moved all of my DNS infra to a bridged VM. Now my DNS will show up with a dedicated IP address on both my LAN (which can be accessed by all of my other containers, my host machine, and any machines connected to my VPN).

I hate doing this. It uses way more resources than reasonable, and as a result I ended up completely replacing the hardware my services are running on - having to use a processor that supports bridging directly to the network interfaces (I needed hardware that specifically used VT-d, otherwise I'd be relying on having to manually route packets - AGAIN).

Maybe there were legitimate reasons locking up port 53 was necessary, but under most circumstances this port should be treated as a system port, and shouldn't ever be remapped by userland software unless it's documented to do so?

@dada513 could you be a little more specific about your use case and what your workaround actually looks like?

• Are you using the hostnames of pods within the same network for DNS resolution? With your solution, I found this still does not work as port 53 gets overwritten. For example, if I have a pod named balancer, but have another pod named dns that maps to the host port 53, I can no longer access any pods by name on sibling container - which makes my balancer pod, and any other pods that talk over the same network - defunct.

• What method are you using for binding? Iptables? Ifconfig? Dnsmasq? Systemd? Via podman?

[d-513]

@martinetd In order to access my my DNS server consistently over both my LAN, but also remotely via WireGuard - I gave up having the DNS port remapped and instead moved all of my DNS infra to a bridged VM. Now my DNS will show up with a dedicated IP address on both my LAN (which can be accessed by all of my other containers, my host machine, and any machines connected to my VPN).

I hate doing this. It uses way more resources than reasonable, and as a result I ended up completely replacing the hardware my services are running on - having to use a processor that supports bridging directly to the network interfaces (I needed hardware that specifically used VT-d, otherwise I'd be relying on having to manually route packets - AGAIN).

Maybe there were legitimate reasons locking up port 53 was necessary, but under most circumstances this port should be treated as a system port, and shouldn't ever be remapped by userland software unless it's documented to do so?

@dada513 could you be a little more specific about your use case and what your workaround actually looks like?

• Are you using the hostnames of pods within the same network for DNS resolution? With your solution, I found this still does not work as port 53 gets overwritten. For example, if I have a pod named balancer, but have another pod named dns that maps to the host port 53, I can no longer access any pods by name on sibling container - which makes my balancer pod, and any other pods that talk over the same network - defunct.

• What method are you using for binding? Iptables? Ifconfig? Dnsmasq? Systemd? Via podman?

Yes, I am using hostnames for DNS resolution. For example in caddy, my reverse proxy, I point to my nextcloud container using its name.
But one major difference is I use containers with a podman network connecting them, and not pods.

I am binding via podman. What I do is use the IP address of the machine directly in port bindings for DNS, like:
192.168.10.250:53:53/udp
This binds it only to the public ip, so podman containers can stil access aavark-dns with the local ip or some podman gateway, while devices can access the actual dns server just fine.

I am too running a wireguard VPN, but I had to set the dns for peers to my machine's public ip

[f1yn]

I had no idea that one could use the full IPv4 as a prefix when describing bindings like that 🤯

I'll give this a try.

[f1yn]

It works*, but I'm skill a little skeptical because I'm noticing that new containers that are added to the shared network seem to have issues resolving on my balancer container. The DNS lookups are returning private ip addresses that are just wrong.

* I'm going to keep monitoring my systems to see if more DNS issues crop up. For now, this seems to solve the issue of putting podman in a completely unusable state like my older comment pointed out.

[martinetd]

I've opened a proof of concept PR in containers/netavark#323 to just use another port -- if the approach is ok for maintainers I'll finish the patch over the next few days.

[Luap99]

I've opened a proof of concept PR in containers/netavark#323 to just use another port -- if the approach is ok for maintainers I'll finish the patch over the next few days.

Well hardcoding to another port does not really solve this. We need to pick a random free port and store it somewhere.

[martinetd]

Well hardcoding to another port does not really solve this. We need to pick a random free port and store it somewhere.

Yes, I've addressed that in the PR -- that's why it's a draft:

I've hardcoded port 1153 as alternative port, we should either use an ephemeral port (problem: we can't know if it's free unless we try to bind to it, so we'd need some retry...) or simpler just make it configurable at network level as a driver-specific option. I'd favor the later.

I'm not spending time to do boring stuff like adding a config item if you don't like the idea in the first place.

[martinetd]

The problem with picking a port at random is that we can't know if it's free, and we need to use it in netavark so we can't just let aardvark-dns try to find one without adding some communication with it (and just retrying to execute aardvark-dns with ports until it works lacks a criteria to decide if it worked, or if it failed because of port binding, or if it failed for something else so we're back to square one)

Adding a config item would allow users which bind something on whatever port we say is default to change it if they want to which would improve the situation considerably from where we are now -- default could actually be 53 if that's what you want and we wouldn't need the iptables rule then, or some arbitrary > 1024 port.
If there are compelling reasons to have netavarak/aardvark-dns automatically find a free port the default value could be that and config value could still be kept for people who'd want to set it explicitly, so it wouldn't be lost anyway.

[Luap99]

Yeah you are right, picking random is not simple at all when trying to fix it properly. We would need to bind with port 0 in aardvark and then somehow communicate the port back. This would also mean that we likely have different ports for each network.

Maybe adding a config option to containers.conf is the best, this would allow user to have a predictable port when they want to use aardvark for other purposes.
Keeping 53 default and have users change the config is acceptable to me. @flouthoc @mheon @baude WDYT?