ExternalName support update request

[brianehlert]

I would like to raise a conversation around supporting ExternalName.
Or rather raise a change to the rather strongly worded statement here.

// ExternalName services can refer to CNAME DNS records that may live // outside of the cluster and as such are difficult to reason about in // terms of conformance. They also may not be safe to forward to (see // CVE-2021-25740 for more information). Implementations SHOULD NOT // support ExternalName Services.

ExternalName support is a common use case among the NGINX Ingress Controller customers (NGINX's project, I can't speak for the K8s community project)
It is combined with other use cases such as egressMTLS, traffic steering, backup service support, and other use cases that make for a very powerful reverse proxy at the edge all up.

What I would like is for the group to consider striking the opinionated language of: "Implementations SHOULD NOT support ExternalName Services."
The remainder of the advice is fine to maintain. But I think blocking a use case that has become relatively common place with Ingress (at least the NGINX implementation) is a bit of overreach.

[howardjohn]

FWIW my intent was to ignore the suggestion (it is SHOULD not MUST). If 3 implementations are planning to do the same (assuming this based on +1s) then it does seem like it makes sense to drop the phrasing.

[arkodg]

in Envoy Gateway we said we won't support it for now envoyproxy/gateway#873
But I prefer if the language here (in upstream) is SHOULD and not MUST as you suggested since in the egress context it is useful.

[robscott]

The current wording in the spec was very much focused on ingress use cases where it seems quite difficult to do safely. The concern of course is that any DNS name can be resolved to sensitive IPs that may expose content the proxy has access to but is not supposed to expose. With the Ingress API, ExternalName Services were never intended as targets, but the spec never said anything to prevent or even recommend against their use. Unfortunately this led to situations where ExternalName Services could be used to expose a lot of sensitive information within the proxy. The most common mitigation I've seen for that within Ingress controllers has been to make the functionality opt-in with a warning of the potential consequences.

In any case, I'd agree with @howardjohn that "SHOULD NOT" is intentionally different than "MUST NOT". Per the language in RFC 2119, "SHOULD NOT" is analogous to "NOT RECOMMENDED". Here's the full text:

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

I think everyone in this thread would generally agree that forwarding to ExternalName Services is risky and should not be supported without careful consideration. That's what we're trying to communicate with the spec, but I know the RFC language may not be obvious here.

Whenever we handle egress in the API we may need to revisit this, but I personally think the spec is appropriately worded right now.

[youngnick]

The big risk with ExternalName is that, by necessity, data planes have to be pretty highly trusted inside a cluster, rendering them vulnerable to confused deputy attacks.

The classic example here is using an ExternalName of localhost (or more importantly, any external name that resolves to localhost, 127.0.0.1, or ::1). If there is anything sensitive running on a port in the data plane pod, allowing ExternalName services effectively grants free access to anyone with Ingress or Gateway access to expose those things. (This caused a CVE for us in Contour, until we moved to domain sockets for Envoy admin operations, which, yes, is a better idea). Also, this problem is not easily fixable with a blocklist or allowlist, because all you need to do is create a localhost.badactor.com entry somewhere that points to a localhost address, and you're back where you were before.

Additionally, because of the way kube DNS works, if you know the name of a service and the name of the namespace it's in, you can use a servicename.svc.namespace.cluster.local ExternalName to expose it, and, unless you have good NetworkPolicy and a CNI that enforces it correctly, the proxy will allow it.

All of which is to say that @robscott and I are in agreement that allowing ExternalName services is a risk, that needs to be carefully managed. This was all pretty surprising to me when we first found it, which is why the recommendation is so strongly worded.

And yes, all of these problems are solvable, and we will need to solve them as part of building Egress support into Gateway API, but until then, the SHOULD NOT stands as a way to recommend that people not do this without being aware what they're getting into.

[brianehlert]

Thanks all for the conversation.
Once it was clear to me about the phrasing and expressly using the RFC language I had a clearer understanding of the intention.
Like many things, this is a "your mileage will vary with your proxy" situation as well as there are opportunities for field sanitization as we have done in our ingress controller to provide some protections.

Thanks all for the conversation.

[dprotaso]

@brianehlert sorta related is EndpointSlices supports a FDQN address type - CoreDNS (1.8.4+) takes this FQDN type and makes it a CNAME entry in the K8s DNS.

SInce Endpoints/Slices are managed by control planes (ie. Kubernetes,Knative) and users don't typically interact with it - thus creating/updating could be restricted for users. I thought this type of resource would be a good candidate as a potential to support (ie. K8s Headless Service with an FQDN EndpointSlice) - #1718

But upstream looking to deprecate it - it seems people were unaware of the CoreDNS support and thought it wasn't used
kubernetes/kubernetes#114210

Could you comment with your ExternalName use cases?

[EltonzHu]

Hi @robscott @dprotaso, one of the use case for using externalName service is Edge CDN back to origin use case.

Use case details: Once the proxy on the edge cluster receives specific user request pattern, it will forward the request to ANOTHER cluster through DNS resolution utilizing the externalName service.

Due to security reasons above, gateway APIs does not support externalName service. Is there any other work around? I know we can utilize endpointslice with FQDN. However, I think this approach has the same security risks as using externalName service.

Any suggestions or work arounds?
Thank you!

[dprotaso]

I know we can utilize endpointslice with FQDN

EndpointSlice with FQDN isn't really defined - can you surface your use case here - kubernetes/kubernetes#114210