How to access rootful container's published port from outside?

[davispuh]

I have rootful container with published port and I can access it from local machine fine but it doesn't work from another machine.
Any ideas how to accomplish this?

From another machine it hangs forever

$ curl http://192.168.1.21:8080/
*   Trying 192.168.1.21:8080...
^C

From same machine where podman container is running works fine

$ curl http://192.168.1.21:8080/
RESPONSE BODY

# podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --shm-size=256M -v /var/lib/ctr/config:/etc/ctr --publish 8080:80 --publish 22:22 docker.io/ctr:latest

$ podman port 28e7841132f5
22/tcp -> 0.0.0.0:22
80/tcp -> 0.0.0.0:8080

$ ss -tln | grep -E '8080|22'
LISTEN 0      512          0.0.0.0:8080      0.0.0.0:*
LISTEN 0      512          0.0.0.0:22        0.0.0.0:*

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 3f:c5:53:5a:b9:a4 brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 192.168.1.21/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe84:4d7b/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether de:9d:de:2c:62:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.1/16 brd 10.88.255.255 scope global podman0
       valid_lft forever preferred_lft forever
    inet6 fe80::dc9d:deff:feab:1432/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
4: veth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman0 state UP group default qlen 1000
    link/ether 12:e8:96:b2:d9:87 brd ff:ff:ff:ff:ff:ff link-netns netns-3d4994cf-56c9-c019-f92b-4515aa91824f
    inet6 fe80::10e8:96ff:fe9f:c1a7/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

$ podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2024-09-24T19:02:27.404596072Z",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

# Generated by iptables-save v1.8.7 on Tue Sep 24 18:54:09 2024
*mangle
:PREROUTING ACCEPT [645:57086]
:INPUT ACCEPT [592:53293]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [591:58589]
:POSTROUTING ACCEPT [591:58589]
COMMIT
# Completed on Tue Sep 24 18:54:09 2024
# Generated by iptables-save v1.8.7 on Tue Sep 24 18:54:09 2024
*raw
:PREROUTING ACCEPT [645:57086]
:OUTPUT ACCEPT [591:58589]
COMMIT
# Completed on Tue Sep 24 18:54:09 2024
# Generated by iptables-save v1.8.7 on Tue Sep 24 18:54:09 2024
*security
:INPUT ACCEPT [580:52357]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [592:58941]
COMMIT
# Completed on Tue Sep 24 18:54:09 2024
# Generated by iptables-save v1.8.7 on Tue Sep 24 18:54:09 2024
*nat
:PREROUTING ACCEPT [66:4789]
:INPUT ACCEPT [1:60]
:OUTPUT ACCEPT [2:120]
:POSTROUTING ACCEPT [2:120]
:NETAVARK-1D8721804F16F - [0:0]
:NETAVARK-DN-1D8721804F16F - [0:0]
:NETAVARK-HOSTPORT-DNAT - [0:0]
:NETAVARK-HOSTPORT-MASQ - [0:0]
:NETAVARK-HOSTPORT-SETMARK - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j NETAVARK-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j NETAVARK-HOSTPORT-DNAT
-A POSTROUTING -j NETAVARK-HOSTPORT-MASQ
-A POSTROUTING -s 10.88.0.0/16 -j NETAVARK-1D8721804F16F
-A NETAVARK-1D8721804F16F -d 10.88.0.0/16 -j ACCEPT
-A NETAVARK-1D8721804F16F ! -d 224.0.0.0/4 -j MASQUERADE
-A NETAVARK-DN-1D8721804F16F -s 10.88.0.0/16 -p tcp -m tcp --dport 22 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -s 127.0.0.1/32 -p tcp -m tcp --dport 22 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.88.0.2:22
-A NETAVARK-DN-1D8721804F16F -s 10.88.0.0/16 -p tcp -m tcp --dport 8080 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -s 127.0.0.1/32 -p tcp -m tcp --dport 8080 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.88.0.2:80
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 22 -m comment --comment "dnat name: podman id: 28e7841132f558610aa9ac91824cdb38a7b3571b91244c09943347001e574625" -j NETAVARK-DN-1D8721804F16F
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8080 -m comment --comment "dnat name: podman id: 28e7841132f558610aa9ac91824cdb38a7b3571b91244c09943347001e574625" -j NETAVARK-DN-1D8721804F16F
-A NETAVARK-HOSTPORT-MASQ -m comment --comment "netavark portfw masq mark" -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A NETAVARK-HOSTPORT-SETMARK -j MARK --set-xmark 0x2000/0x2000
COMMIT
# Completed on Tue Sep 24 18:54:09 2024
# Generated by iptables-save v1.8.7 on Tue Sep 24 18:54:09 2024
*filter
:INPUT ACCEPT [599:53657]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [605:62425]
:NETAVARK_FORWARD - [0:0]
:NETAVARK_INPUT - [0:0]
:NETAVARK_ISOLATION_2 - [0:0]
:NETAVARK_ISOLATION_3 - [0:0]
-A INPUT -m comment --comment "netavark firewall rules" -j NETAVARK_INPUT
-A FORWARD -m comment --comment "netavark firewall rules" -j NETAVARK_FORWARD
-A NETAVARK_FORWARD -m conntrack --ctstate INVALID -j DROP
-A NETAVARK_FORWARD -d 10.88.0.0/16 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETAVARK_FORWARD -s 10.88.0.0/16 -j ACCEPT
-A NETAVARK_INPUT -s 10.88.0.0/16 -p udp -m udp --dport 53 -j ACCEPT
-A NETAVARK_ISOLATION_3 -o podman0 -j DROP
-A NETAVARK_ISOLATION_3 -j NETAVARK_ISOLATION_2
COMMIT
# Completed on Tue Sep 24 18:54:09 2024

I'm on openSUSE 15.6 with podman 4.9.5 and I disabled firewalld to rule it out.

[davispuh]

Something doesn't work with container's network because from inside container I can't access internet but I can access host with 10.88.0.1

[rhatdan]

Did you check the firewall?

[davispuh]

Yeah I disabled firewalld on both machines

[davispuh]

It was something really strange.

I cleared nft rules with nft flush ruleset and rebooted host and container and now it started working.

[rhatdan]

Sometimes iptables rules can get mucked up based on package updates.

[davispuh]

It's super strange, now I rebooted again and it stopped working. Still no idea what made it work for a moment before.

[davispuh]

This was such a PITA to find out, but basically podman container was starting before firewalld had started so only iptables was setup but later when firewalld started it switched over to nftables but those doesn't have rules for container.
I fixed this by adding After=firewalld.service in container systemd unit and now it always works.

[rhatdan]

Are you using quadlets or the podman.service?

[davispuh]

I'm using quadlet, it looks like this

[Unit]
Description=Container
After=local-fs.target firewalld.service

[Container]
Image=docker.io/ctr:latest
PublishPort=0.0.0.0:8080:80
PublishPort=0.0.0.0:22:22
Volume=/var/lib/ctr/config:/etc/ctr
ShmSize=256M
AutoUpdate=registry

[Install]
WantedBy=multi-user.target default.target