iptables rules flushed too eagerly from NETAVARK_FORWARD chain, breaking container networking

[nitzmahone]

Somewhere between netavark 1.0.2 and 1.2.0, netavark started flushing rules from the NETAVARK_FORWARD chain every time a container exits (instead of only for the last container on a given network). Starting another container on that network reinstates the rules as expected, but as soon as any container using that network exits, they're flushed again, trashing intra-container connectivity on that network. This issue is masked by the default FORWARD chain policy of ACCEPT on, eg, Fedora/RHEL (and further that iptables is bypassed by default), but running some packages on an empty default iptables config (eg, moby-engine) will change that policy to DROP and force bridge devices through iptables, which exposes the problem very clearly.

The following command reproduces the issue and shows the problematic chain flush:

iptables -L NETAVARK_FORWARD -n; \
podman run -d alpine:latest sleep 1000; \
iptables -L NETAVARK_FORWARD -n; \
podman run alpine:latest echo "i am a container that is exiting..."; \
iptables -L NETAVARK_FORWARD -n; \
podman run -d alpine:latest sleep 1000; \
iptables -L NETAVARK_FORWARD -n

With podman 4.3.0 and netavark 1.2.0 from Fedora 36, we see the broken behavior:

Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
60b18c8713d9fe04378afb52d423a5c642c20e314ba3e745347fcf8b9b60be95
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.0/16         0.0.0.0/0           
i am a container that is exiting...
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ee1928545063fee05c9118a10d52e24457dc0ece659e7a152168bf7214c45aa3
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.0/16         0.0.0.0/0    

(note that the chain has been flushed of all rules for the default network after a container has exited, but that they reappear as soon as another container has started on that network, or on eg podman network reload --all)

After rolling back to netavark 1.0.2 and running the same test command, we see the expected behavior (the rules stay in the NETAVARK_FORWARD chain):

Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
2480224fc4f04caf01c1a054cede414739fdd3fd6d80372fb2c59994e7c990e1
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.0/16         0.0.0.0/0           
i am a container that is exiting...
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.0/16         0.0.0.0/0           
cf5c6b01ef5f473b95340bd262e0e8f6dc59859dd248769b18d1198743defd2f
Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.0/16         0.0.0.0/0      

[Luap99]

I take a look