Egress supports except property to allow noSNAT for assigned destination

[leonstack]

Describe what you are trying to solve
My environment has multi-nic per-node for several network(such as management, monitor, public), and there are two applications in my environment. One application uses antrea(called antrea-app) and another uses hostNetwork(called host-app) . The antrea-app wants to access the external network and also wants to access the host-app(such as monitor network).
If the egressIP function is used for the antrea-app, no matter the destination is the external network or host-app, the packets will always be sent to the node where the egressIP belong to, and then the packets will be sent according to the host route of the node, which will hugely reduce the network performance between antrea-app and host-app.

Describe the solution you have in mind
Add a new property named except to Egress, means if destination matchs except value, these packets won't be sent to the node which egressIP belong to.
After adding the except field, and set the value to the IP or CIDR where host-app listening(such as monitor network). When the egressIP is used for antrea-app, if the destination address is host-app, the packets will no longer be sent to the node where the egressIP belong to, but sent directly from the current node and communicate with host-app directly. If the destination address doesn't match the except, the packets is still sent to the node where egressIP belong to.

[tnqn]

@leonstack Thanks for the proposal. Supporting excluding specific traffic from being SNATed sounds reasonable to me. Besides your case, I can imagine that if there are BMs or VMs running in the same network as the Kubernetes cluster, user may want their Pod talk to them directly. So having except can help that case too.

@jianjuns @antoninbas @wenqiq Could you share your opinions if any?

[antoninbas]

This sounds good to me. One other issue (#2671) asked for the ability to filter by protocol (TCP / UDP), so maybe we can think of a generic traffic filtering mechanism that covers multiple use cases. I am not totally sure why one would want to filter by protocol though, so maybe @eranshpiner could provide more information about his use case.

[leonstack]

hi @tnqn @antoninbas , seems the except should include [cidr, protocol, port] for more situation.
Or we can focus on [cidr, protocol] that I can't think of a use case for port.

[tnqn]

@leonstack I cannot think of a use case that different protocols and ports expect different egress behavior either. But we can make the API extensible even we only support cidr for now. Something like the following should work:

@@ -226,6 +226,12 @@ type EgressSpec struct {
        // If it is non-empty, the EgressIP will be assigned to a Node specified by the pool automatically and will failover
        // to a different Node when the Node becomes unreachable.
        ExternalIPPool string `json:"externalIPPool"`
+
+       Excepts []Except `json:"excepts,omitempty"`
+}
+
+type Except struct {
+       CIDR string
 }

[jianjuns]

Just wonder do we have requirements for per Egress "except", or we just need a global list of excluded CIDRs.

[leonstack]

Just wonder do we have requirements for per Egress "except", or we just need a global list of excluded CIDRs.

Global list can solve my problem, but for #2671, I'm not sure it's flexible enough.

[tnqn]

I agree with @jianjuns that these except CIDRs are normally global in a cluster. It seems it doesn't make a lot of sense that an IP is an external address to one Pod but an internal address to another. And repeating the configuration in all Egress objects will be boring. Since we haven't received any feedback for #2671, should we just do the way that makes most sense for now? Maybe an option group for egress feature like this?

egress:
    exceptCIDRs: x.x.x.x/y, ...

It seems it would simplify the implementation too as we only need to classify these except CIDRs only once, instead of once per egress.
What you think?

[leonstack]

I agree with @jianjuns that these except CIDRs are normally global in a cluster. It seems it doesn't make a lot of sense that an IP is an external address to one Pod but an internal address to another. And repeating the configuration in all Egress objects will be boring. Since we haven't received any feedback for #2671, should we just do the way that makes most sense for now? Maybe an option group for egress feature like this?

egress:
    exceptCIDRs: x.x.x.x/y, ...

It seems it would simplify the implementation too as we only need to classify these except CIDRs only once, instead of once per egress.
What you think?

Agreed, and the implementation will be more simplified, we can ignore the #2617 that I haven't found a case avoid SNAT by protocol filter.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days