Pod IPs removed from EndpointSlices for non-headless services in v0.15

[t0lya]

I have a question regarding Aggregate Service Import enhancement proposal which was implemented in #1144 and shipped in Submariner release 0.15. This change removed pod IP endpoints from EndpointSlice documents for non-headless ClusterIP services.

E.g., Endpoints list contains ClusterIP (address 10.0.0.1 in example). Before 0.15, endpoints contained list of pod IPs.

addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - 10.0.1.1
  conditions:
    ready: true
kind: EndpointSlice
metadata:
  labels:
    endpointslice.kubernetes.io/managed-by: lighthouse-agent.submariner.io
    lighthouse.submariner.io/is-headless: "false"
    lighthouse.submariner.io/sourceNamespace: default
    multicluster.kubernetes.io/service-name: prometheus-node-exporter
    multicluster.kubernetes.io/source-cluster: cluster-a
    submariner-io/clusterID:  cluster-a
    submariner-io/originatingNamespace: default
  name: prometheus-node-exporter-default-cluster-a
  namespace: submariner-k8s-broker
ports:
- name: metrics
  port: 9100
  protocol: TCP

What was the reasoning behind this change and why is it needed? I could not find the reasoning while reading through Aggregate Service Import enhancement proposal. I do not see such a requirement on EndpointSlices in KEP 1645.

My team has a use-case for getting the list of ready endpoints for a given ClusterIP service across all connected clusters. We currently implement this by querying the ready endpoints in the EndpointSlice document on the broker cluster. We are trying to figure out how to preserve this use-case while upgrading Submariner to 0.15.

[tpantelis]

The purpose of the aggregated ServiceImport changes was to align Submariner with the MCS spec (KEP 1645). Prior, each cluster's service IP (for non-headless) was published in its own ServiceImport. Now they're published via a Submariner-specific EndpointSlice. We did previously publish an EndpointSlice but that was only for the purpose of communicating to the CoreDNS plugin load balancer whether or not the service has any backing pod endpoints. We could have just published an EndpointSlice with no endpoint addresses.

The EndpointSlice is really an implementation detail and was/is not intended for consumption outside of Submariner. The fact that previously the EndpointSlice contained the pod endpoints was really just happenstance. While the MCS spec does talk about the use of EndpointSlices to communicate endpoints, it does not require it, in fact a recent discussion in the Multi-Cluster SIG proposes to remove any mention of EndpointSlices in the spec since one implementation doesn't use them at all.

[abguptms]

We have varied customer requirements:

1. Directly use the in-built DNS (clusterset.local) and proxy-based routing. This is being worked with the cloud provider to support direct connections between clusters in the same virtual network. For this, we have dependency on coredns plugin. Currently, it is being used by our customers with headless services since they resolve Pod IPs and that connectivity works.
2. Customized load balancing solutions which require pod IPs to be moved around between clusters to allow different routing strategies (such as weighted). This is where the 0.15 breaks our current implementations.

Our primary use for Submariner is discovery of both ServiceIPs and PodIPs. Routing/connectivity component is not something we plan to use since our scenarios allow us to access IPs directly between different clusters.

We can definitely look toward contributing and adding optional behavior. Is the non headless clusterIP in endpointslice (exported ), expected to be used for DNS resolution for non-headless services too? Isn't ServiceImport document the reliable source for it?

[vthapar]

I’m sure when EndpointSlice was designed, they weren’t thinking of multicluster use cases. A multicluster deployment has different set of scale and performance requirements than a single cluster. Ideally there should’ve been something like MultiClusterEndpointSlice for this.

ServiceImport vs EndpointSlice is an implementation detail. ServiceImport is supposed to contain the VirtualIP for multiclusterservice while endpointslice is supposed to include IPs to route traffic for that service, similar to a single cluster Service and endpointSlice objects. But in LH we do it differently. We don't use Virtual IP so nothing to store in ServiceImport, and EndpointSlices contain ServiceIPs for that service in different clusters. This way, routing in local cluster only needs to direct traffic to relevant cluster and then kube-proxy+cni on that cluster will loadbalance it to one of the pods backing. This saves information we need to sync across clusters and puts the onus of directing traffic to specific backend pod on that cluster's kube-proxy+CNI. One advantage of this approach is that decision of which pod to go to is taken at last hop. With other approach where it is done at first hop, it is possible that remote pod selected by local cluster is no longer available because of time delay in syncing information across clusters.

[tpantelis]

Our primary use for Submariner is discovery of both ServiceIPs and PodIPs. Routing/connectivity component is not something we plan to use since our scenarios allow us to access IPs directly between different clusters.

OK - that's fine.

We can definitely look toward contributing and adding optional behavior. Is the non headless clusterIP in endpointslice (exported ), expected to be used for DNS resolution for non-headless services too? Isn't ServiceImport document the reliable source for it?

The ServiceImport is supposed to contain a clusterset VIP and be clusterset-wide and live in the service namespace on each cluster. We've now satisfied the latter part but we still don't set a clusterset VIP, perhaps at some point. We decided to use an EndpointSlice to publish each cluster's service IP to align with the MCS KEP direction and that is used by our CoreDNS plugin for resolution. Hope that answers your question.

With the optional behavior, we could add an annotation to the EndpointSlice indicating it contains pod endpoints so that the CoreDNS plugin ignores it.

[tpantelis]

@abguptms Is this still a requirement for you? If so we should convert this back to an issue.

I think a fairly simple and clean solution would be to use the "endpointslice.kubernetes.io/managed-by" label to control the behavior of the generated EndpointSlices. So you would add the "endpointslice.kubernetes.io/managed-by" label with a custom value to the ServiceExport. This would signal the lighthouse EndpointSlice controller to publish the endpoints from the K8s EndpointSlice as is with the custom "endpointslice.kubernetes.io/managed-by" label and sync to the broker. With the custom label value it wouldn't interfere with K8s or lighthouse. Note this would only apply to non-headless services as we already publish the pod endpoints for headless services.

[abguptms]

@tpantelis, yes, it is still a requirement for us. Thanks for confirming. We have done a quick workaround currently (business needs) to setup a headless app and get the endpointslices to make sure discovery is not broken for us. I agree with the solution provided by you above. We can definitely setup a custom managed-by label to the ServiceExport for lighthouse to export our endpoints always.

[tpantelis]

OK I'll convert it. I might have cycles to work on this for next release. It should be fairly straightforward. I assume you would need the EndpointSlices exported to the broker but what about distributed to the other clusters?

[abguptms]

My apologies, I thought that was assumed. Yes, we would need both export and import of endpointslices. This means, all connected clusters should be able to receive these from the broker. This will help us generate an exhaustive list of endpoints across the connected set for internal load balancing.

[tpantelis]

That's what I figured - just wanted to verify.