Add support for UDPRoute API

[arkodg]

Description:
Add Support for the UDP Route API defined here

Related #47

[arkodg]

we should wait on #413 which will refactor the provider before starting this issue

[zhaohuabing]

@arkodg May I take this one? It would be a relatively easy start for me to dig into the core mechanism of EG since I can refer to the implementation of the current HTTRoute. If this one goes well, I would like to take #642 as well.

[arkodg]

go for it @zhaohuabing, suggest raising incremental PRs in the below order to increase PR review speeds and reduce merge conflicts :)

• Xds IR
• Xds Translator
• Gateway API Translator
• Provider (which is currently being refactored)

[youngnick]

I'd like to see some work done on how the UDP flows will actually work. It's pretty straightforward to set up a UDP listener for Envoy, but it's very difficult to preserve the client address at the same time.

Basically, you have two options:

• Use Envoy in non-transparent proxy mode, which means that the client address will be seen at the backend pods as Envoy's address. As I said, most UDP protocols don't like this.
• Use Envoy in transparent proxy mode, which means that packets will be forwarded to the backend pods without changing the client IP, but the server IP will also not be updated to Envoy's, which means that the return path from the backend back to the client will both not go via Envoy, and will be from a different IP address than what the packet was sent to (since it will be the backend's address). Most firewalls will drop UDP traffic like this, that both doesn't come from and address traffic is already heading to, and (most likely) comes from an RFC1918 address.

In particular, on Contour, we've been asked to have Contour handle SIP traffic, and SIP in particular is a very bad fit for this sort of proxying, as the RTP used for the actual data streams (as opposed to the control traffic) is very dynamic and difficult to proxy correctly.

I'm not opposed to doing UDPRoute in Envoy Gateway, but I would like to see significantly more design around how to make it available in a way that meets user expectations. Just adding support for the existing UDP modes in Envoy does not match up to what users will expect when you add UDP forwarding.

[zhaohuabing]

I think transparent proxy mode can be benifical for both TCP and UDP since in some use cases, the server behind the EG would like to know the original addreess of the client. To get this done, we'll need to set the route inside a cluster to get the return traffic route back to the EG, and then EG can send the traffic right back to the client.

[arkodg]

@youngnick the https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/udp_filters/udp_proxy docs states that envoy operates as a non transparent proxy.
The xds implementation does not seem too complex, if you prefer, either @zhaohuabing or I could
PoC with Envoy proxying UDP traffic to a pod running a TFTP server or UDP Ping server and share those details on this issue.
If some UDP protocols dont like non transparent mode , we will have have add those limitations to the EG docs for now.

[zhaohuabing]

non-transparent proxy should work, I'll test that after the translators implemention is done. transparent proxy mode is also helpful, maybe we could discuss how to support it if there're more requests asking for it from the clients. It looks like we need to introduce new components if we want to addd transparent mode, we will need a CNI to set up the route. Besides, we should also consider the transparent mode for TCP listeners together.

[arkodg]

Reread the envoy docs which states that envoy also support transparent mode but needs added privileges ( not given to the envoy deployment today )
@youngnick do you have any info handy on which protocols support non transparent mode and which dont ?

[youngnick]

It's tricky, but in general UDP protocols expect the client address to be the actual one. DNS, and syslog are the ones I can think of from the top of my head. I'm not sure about other ones - I know that lots of gaming applications use UDP as well. It's going to be on a per-protocol basis though - and lots of people roll their own protocol.

If we really want to do it, it will be extremely important to call out the no-client-address limitations of proxied mode, definitely.

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy - I don't feel like us building that into Envoy Gateway is a very good idea without doing a pretty detailed design first.

[danehans]

xref #646 for adding UDPRoute to IR.

[danehans]

Use Envoy in non-transparent proxy mode, which means that the client address will be seen at the backend pods as Envoy's address. As I said, most UDP protocols don't like this.

Although most UDP protocols don't like this, non-transparent proxy mode could be considered "initial" UDPRoute support. We can document these details while we work on a long-term UDPRoute solution that covers the other UDP protocols.

@mattklein123 PTAL at this issue and #643. Let us know if you have any input on adding L4 support.

[danehans]

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy

Maybe a solution similar to kube-proxy's internalTrafficPolicy: Local can help workaround this issue?

[arkodg]

cc @moderation

[youngnick]

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy

Maybe a solution similar to kube-proxy's internalTrafficPolicy: Local can help workaround this issue?

Sadly, it's not about the inbound traffic, transparent mode will get the packets to the backend properly. It's about the return path from the backend back to the client - because Envoy doesn't do any address translation, the packets will be sent straight from the backend to the client, without going back through Envoy. To change that behavior, you'd need to mess with the routing tables for the pod, basically making Envoy the default gateway for all IP traffic in the pod (this is how traditional load balancers like F5 work, they're almost always the default gateway for the segment where the backends reside).

[moderation]

@youngnick a couple of questions:

1. NGINX in theory supports UDP ingress today irrespective of Envoy Gateway - do they have a feature or implementation that differs from Envoy that enables them to provide this UDP ingress solution?
2. Ambassador Labs Emissary Ingress which is based on Envoy supports HTTP/3 - https://blog.getambassador.io/how-to-implement-http3-support-with-ambassador-edge-stack-c57bae385a88. HTTP/3 rides on QUIC which is UDP based. What are they doing that is out of reach for Envoy Gateway supporting UDP?
3. Should the documentation of the Kubernetes Gateway API specification modify their documentation to indicate that of the reverse proxies (NGINX, HAProxy, Traefik, Caddy etc.), Envoy is unable to support UDPRoute?

[youngnick]

@youngnick a couple of questions:

1. NGINX in theory supports UDP ingress today irrespective of Envoy Gateway - do they have a feature or implementation that differs from Envoy that enables them to provide this UDP ingress solution

No, from what I can see, their UDP implementation will have the same client address limitation. I seem to recall looking at a version of their documentation that called this out, but I can't see it in their current one.

In addition, the fact that their Service Mesh product uses eBPF to inject PROXY protocol headers in order to make this work tends to bear this out, for me.

2. Ambassador Labs Emissary Ingress which is based on Envoy supports HTTP/3. HTTP/3 rides on QUIC which is UDP based. What are they doing that is out of reach for Envoy Gateway supporting UDP?

QUIC is different, because QUIC uses UDP as a transport, but has higher-layer constructs to allow the backend to rebuild the actual client address (like X-Forwarded-For in HTTP/1 and HTTP/2). So it doesn't matter in those cases if the actual client IP is lost along the way.

Most other UDP protocols don't have this, sadly.

3. Should the documentation of the Kubernetes Gateway API specification modify their documentation to indicate that of the reverse proxies, Envoy is unable to support UDPRoute?

It's not really a matter for Gateway API - Gateway API defines what a UDPRoute is - the fact that Envoy running as an in-cluster proxy can't do it isn't really relevant there.

I would love to have a way to make this work the way we all want it to, but trying to proxy UDP with cluster networking really is the worst-case scenario, sadly.

[zhaohuabing]

It seems like we don't have a workable solution for trasparent proxying for both UDP and TCP in the short term. So I'll go with the non-trasparent mode and add some documentation on that to make it clear to the users. Does this sound good?

BTY, if people really relies on the real source address in an application protocol based on UDP and want to deploy their applications in cloud infrastructure, they may need to switch to an application-level solution to get the source address because there're good chances that there're other non-transparent hops between the client and the server.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[arkodg]

lets keep this issue open until end to end functionality is shown in the docs

[danehans]

@zhaohuabing do you mind adding a user doc to close out this issue?

[zhaohuabing]

@danehans Sorry for the delay. My hands were tide for the last few weeks. I'll add the doc this week and close this issue.