Allow cross namespace references

[howardjohn]

What would you like to be added:
Cross namespaces references in Route and BackendPolicy.

Examples:

A gateway may want to expose a route for a service in another namespace. For example, maybe we set up a dedicated gateway namespace a define a few routes to expose Services in other namespaces. This may sound like a security issue (a user can expose my secret Service externally without my permission), but it isn't inherently so - a user could already do this by manually configuring the proxy. If the user actually wants to restrict traffic from other namespaces, like our gateway namespace here, they need to do that in NetworkPolicy or similar, not assume someone won't proxy external traffic to their service:

apiVersion: networking.x-k8s.io/v1alpha1
kind: BackendPolicy
metadata:
  name: policy
  namespace: gateway
spec:
  backendRefs:
  - name: foo
    kind: service
    namespace: bar # NEW: cross namespace reference
    port: 80
---
apiVersion: networking.x-k8s.io/v1alpha1
kind: HTTPRoute
metadata:
  name: foo-route
  namespace: gateway
spec:
  hostnames:
  - foo.example.com
  rules:
  - forwardTo:
    - serviceName: foo
      namespace: bar
      port: 80

A user may migrate to a new namespace and need to do a traffic split

apiVersion: networking.x-k8s.io/v1alpha1
kind: HTTPRoute
metadata:
  name: foo-route
  namespace: gateway
spec:
  hostnames:
  - foo.example.com
  rules:
  - forwardTo:
    - serviceName: foo
      namespace: bar
      port: 80
      weight: 2
    - serviceName: foo-new
      namespace: bar-new
      port: 80
      weight: 1

[robscott]

I think the cross namespace traffic-splitting is a pretty compelling use case. This actually feels somewhat related to #573 (cc @nak3).

My concern is that it risks making the API too complex. We already provide one place in the API for cross-namespace references (Gateway->Route). Adding a second cross-namespace reference (Route->Backend) could potentially become confusing.

With the Gateway->Route relationship, we added a two way handshake that required both the Route and Gateway to effectively opt-in to references from/to a different namespace. In the Route->Backend relationship, we don't control both resources, so we couldn't achieve the same kind of handshake. Instead it seems like our best way to limit this functionality would be some kind of opt-in on either the Gateway or GatewayClass resource.

If we can find a way to reasonably lock this down + document the use case well, I think this could be compelling. Interested in what others think here.

/cc @bowei @danehans @hbagdi

[bowei]

The key here is to understand what the permissions model is -- and how this will be enforced.

The nice thing with same namespace is that it is prevented by the syntax of the API itself -- there is simply no way for the user to create an invalid configuration -- we don't need to call out to controller behavior to give guarrantees.

[hbagdi]

a user could already do this by manually configuring the proxy. If the user actually wants to restrict traffic from other namespaces, like our gateway namespace here, they need to do that in NetworkPolicy or similar, not assume someone won't proxy external traffic to their service

I don't completely agree with this reasoning. Cross-namespace problems in k8s are present in a lot of places and network is no exception. The isolation doesn't guarantee security at runtime but it does guarantee security at configuration time, which is plenty useful and adding such features should be an exception rather than the norm. To me, the use-case of migrating between namespaces doesn't seem strong enough to warrant an exception (yet).

[robscott]

I think this may be less big of a deal than I'd originally thought. I started to sketch out some ways we could do this with some kind of two way handshake, but that felt like we'd be providing a false sense of security. As we've already noted, there's absolutely no boundary to prevent requests from reaching services in other namespaces, so we were really only talking about a referential boundary here. But then the example of ExternalName Service came up:

apiVersion: v1
kind: Service
metadata:
  name: test-service
  namespace: default
spec:
  type: ExternalName
  externalName: example.prod.svc.cluster.local

This does exactly what it looks like, it proxies requests to a Service in another namespace. So if that's already possible, it makes me wonder how important or meaningful the boundary really is here.

With the Gateway -> Route relationship, we're letting each role define the namespace(s) they trust to configure connected Gateways or Routes. I think that still has clear value since we can fully enforce those boundaries.

On the other hand, I'm not sure we can add a namespace boundary for routing to a Service where one simply doesn't exist. If we had a boundary with easy workarounds such as an ExternalName Service, that boundary likely does more harm than good as it could give users a false sense of security.

[robscott]

Closely related to #634

[banks]

Came to open the same discussion and found this which already covers a lot of what I was thinking.

I think the first use-case presented where org-wide gateways are managed in a central namespace is important.

We have user requirements for a centralised NetOps team to own the gateway and operate it in one namespace, while being able to control which HTTP path prefixes route to which backend services which may be owned by different teams in different namespaces. For example:

• www.example.com/billing should map to the billing service in billing namespace
• www.example.com/users should map to the user service in user namespace

That's pretty much the use-case @howardjohn described first and it's not really possible currently as far as I can see.

The example on the API website is actually pretty similar except that the actual prefix management is done by the "Developer" roles within their namespace rather than by the "Cluster Operator" which is effectively what some of our users are requesting.

Currently the NetOps team would need RBAC permissions to create HTTPRoutes in each service namespace to setup the prefix forwarding. They'd then need to use RBAC rules or likely a custom admission controller policy to prevent the application teams from modifying those and affecting the prefix their service is exposed at or interfering with other team's prefixes etc.

So far the Gateway API's emphasis on role-based config has been a great match for the use-cases we've uncovered during research, but this seems to be one area where it doesn't quite match up for those users who prefer to manage all routing centrally, while routing traffic to application teams in different namespaces.

On the other hand, I'm not sure we can add a namespace boundary for routing to a Service where one simply doesn't exist. If we had a boundary with easy workarounds such as an ExternalName Service, that boundary likely does more harm than good as it could give users a false sense of security.

I wasn't quite clear if this was arguing for or against allowing ForwardTo to specify cross-boundary services?

Is the suggestion that ExternalName is already a workaround so we don't need another way, or that since ExternalName workaround already exists, the API might as well provide the equivalent with a better UX for this use-case?

Really happy to share more if it's useful!

[robscott]

Thanks for the great feedback here @banks! I think your example of centrally managed Gateway + Routing is going to be fairly common.

Currently the NetOps team would need RBAC permissions to create HTTPRoutes in each service namespace to setup the prefix forwarding. They'd then need to use RBAC rules or likely a custom admission controller policy to prevent the application teams from modifying those and affecting the prefix their service is exposed at or interfering with other team's prefixes etc.

This is closely related to #634 which would enable further Routing in the target namespace through Route delegation. Interested in your feedback on that approach as well.

Is the suggestion that ExternalName is already a workaround so we don't need another way, or that since ExternalName workaround already exists, the API might as well provide the equivalent with a better UX for this use-case?

Yeah, this was pretty confusing. I was trying to say that since it's already possible, we might as well make it obvious that it's possible instead of hiding the functionality. And if that's our goal, it probably makes the most sense just to add a namespace ref to the forwardTo instead of requiring users to use the ExternalName workaround.

This is really top of mind for us over the next couple weeks. This will actually be our main topic for our next community meeting.

[banks]

@robscott thanks for clarifying - that reasoning makes sense to me.

Yes #634 is closely related and even goes further towards a model we've had requests for where there is central control of some portion of the routing decision (e.g. path prefix) while other routing rules (e.g. rest of path, headers, query params) can still be delegated to application teams. I'll follow that issue closely too.