Ability to specify DNAT destination when using multiple bridges and publishing ports

[JSkold]

The issue

A container using multiple bridge networks and port publish will not have consistent destination addresses to the container.

See this netavark DNAT rule, for every port and network a rule is added:

Chain NETAVARK-HOSTPORT-DNAT (2 references)
 pkts bytes target                   prot opt in     out     source               destination
 0    0     NETAVARK-DN-{net1_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:http 
 0    0     NETAVARK-DN-{net1_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:https 
 0    0     NETAVARK-DN-{net2_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:http 
 0    0     NETAVARK-DN-{net2_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:https 

note: NETAVARK-DN-{net_hash} target does the actual DNAT.

Since the rule matching is the same for both networks all traffic to the host port will be DNATed to the first network. The order in which these rules are added seems to be random with each container creation and since there is no way for the user to influence this, it creates problems when you want to treat these networks differently.

Simply recreating another container with the same networks yielded these rules:

Chain NETAVARK-HOSTPORT-DNAT (2 references)
 pkts bytes target                   prot opt in     out     source               destination
 0    0     NETAVARK-DN-{net2_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:http 
 0    0     NETAVARK-DN-{net2_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:https 
 0    0     NETAVARK-DN-{net1_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:http 
 0    0     NETAVARK-DN-{net1_hash}  tcp  --  any    any     anywhere             anywhere           tcp dpt:https 

Note the reversed order of the networks.

Proposed solution

Allow the user to specify which bridge network a port mapping should apply to. When publishing a port, an additional option is avaliable which specifies what interface external traffic is DNATed to:

--publish, -p=[[ip:][hostPort]:]containerPort[/protocol][,bridge]

# Example:
--network bridge01 --network bridge02 --publish 127.0.0.1:80:80/tcp,bridge02 --publish 443,bridge01

Since only the first DNAT rules from the above examples actually have any effect, the rules would look something like this instead:

Chain NETAVARK-HOSTPORT-DNAT (2 references)
 pkts bytes target                       prot opt in     out     source           destination
 0    0     NETAVARK-DN-{bridge02_hash}  tcp  --  any    any     anywhere         anywhere           tcp dpt:http 
 0    0     NETAVARK-DN-{bridge01_hash}  tcp  --  any    any     anywhere         anywhere           tcp dpt:https 

The bridge option would be optional in order to provide backwards compatibility, when not specified the destination bridge would be random.

I think this could be an option field in the PortBindings objects, netavark would then only create the DNAT rules for that network.

Most of these changes would be in netavark I reckon, but I would like to start the discussion here to see if the frontend change is acceptable? I haven't used the rootless networking modes so I'm not sure how it would affect them. I'm willing to attempt to implement this myself and submit a PR.