linkerd-multicluster remoteDiscoverySelector default selects all remote services

[willhughes-au]

What is the issue?

remoteDiscoverySelector was introduced on the link-crd for linkerd-multicluster in stable-2.14.

Added a new remoteDiscoverySelector field to the multicluster Link CRD,
which enables a service mirroring mode where the control plane
performs discovery for the mirrored service from the remote cluster, rather
than creating Endpoints for the mirrored service in the source cluster

Unfortunately the default behavior is that it selects all remote services.

How can it be reproduced?

Setup two K8S clusters, clusterA and clusterB with Linkerd 2.13.6 installed and working with linkerd-multicluster.

On clusterB have at least two services. Add a label on one service with 'mirror.linkerd.io/mirror-to-clusterA'=true

On clusterA add a Link resource that looks like:

apiVersion: multicluster.linkerd.io/v1alpha1
kind: Link
metadata:
  name: clusterB
  namespace: linkerd-multicluster
spec:
  clusterCredentialsSecret: cluster-credentials-clusterB
  gatewayAddress: >-
    the-gateway-address-for-clusterB.example.com
  gatewayIdentity: >-
    the-gateway-identity-for-clusterB.cluster.local
  gatewayPort: '4143'
  probeSpec:
    path: /ready
    period: 3s
    port: '4191'
  selector:
    matchLabels:
      mirror.linkerd.io/mirror-to-clusterA: 'true'
  targetClusterDomain: cluster.local
  targetClusterLinkerdNamespace: linkerd
  targetClusterName: clusterB

Make sure the link is all working, and that the service from clusterB appears on clusterA.

Now upgrade clusterA to 2.14, including linkerd-multicluster.

Recreate the link.

Observe that all the services from clusterB are mirrored to clusterA.

To stop this behavior, you have to set the remoteDiscoverySelector to something that is not null or an empty set.

eg:

  remoteDiscoverySelector:
    matchExpressions: []
    matchLabels:
      mirror.linkerd.io/stop-with-the-remote-discovery: 'true'

Logs, error output, etc

I'm not sure what relevant logs there are.

The pod linkerd-service-mirror-clusterB on clusterA is filled with a whole lot of events about creating a new service mirror for

output of linkerd check -o short

$ linkerd check -o short --context clusterA
linkerd-multicluster
--------------------
× all mirror services have endpoints
    Some mirror services do not have endpoints:
    (list of services that were picked up automatically)
    see https://linkerd.io/2.14/checks/#l5d-multicluster-services-endpoints for hints

Status check results are ×

Environment

• Kubernetes Version: 1.27
• Cluster Environment: AWS EKS
• Host OS: Amazon Linux 2, EKS Optimised version
• Linkerd: stable-2.14

Possible solution

If remoteDiscoverySelector is not set on the Link resource, remote discovery should not be used.

Additional context

No response

Would you like to work on fixing this bug?

no

[calvinbui]

same thing happening to us. if the namespace exists in both clusters, it'll mirror the service

Untitled.mp4

[mateiidavid]

Hi folks! Thank you for raising this. I attempted to reproduce and I did not manage to. I followed @willhughes-au's instructions.

1. Install stable-2.13.6 and add a Link with a custom selector. In my cluster B (target) I added two services:

:; k get svc --context=k3d-target
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP   10m
nginx-svc    ClusterIP   10.43.48.141   <none>        80/TCP    6m43s
podinfo      ClusterIP   10.43.80.123   <none>        80/TCP    6m54s

My link resource:

  probeSpec:
    path: /ready
    period: 3s
    port: "4191"
  selector:
    matchLabels:
      mirror.linkerd.io/mirror-to-clusterA: "true"
  targetClusterDomain: cluster.local
  targetClusterLinkerdNamespace: linkerd
  targetClusterName: target

Then annotated podinfo:

:; k get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes       ClusterIP   10.43.0.1       <none>        443/TCP   10m
podinfo-target   ClusterIP   10.43.117.240   <none>        80/TCP    6m39s

2. Upgrade Linkerd, including multicluster.
3. Re-link

:; linkerd --context=k3d-target mc link --cluster-name target --api-server-address="https://192.168.224.2:6443" --selector=mirror.linkerd.io/mirror-to-clusterA=true | k --context=k3d-source apply -f -

  clusterCredentialsSecret: cluster-credentials-target
  gatewayAddress: 192.168.224.2
  gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
  gatewayPort: "4143"
  probeSpec:
    path: /ready
    period: 3s
    port: "4191"
  remoteDiscoverySelector:
    matchLabels:
      mirror.linkerd.io/exported: remote-discovery
  selector:
    matchLabels:
      mirror.linkerd.io/mirror-to-clusterA: "true"
  targetClusterDomain: cluster.local
  targetClusterLinkerdNamespace: linkerd
  targetClusterName: target

Only podinfo-target is mirrored, as expected. I would have expected based on the description to see both of my services mirrored, that's not the case though. Am I misunderstanding the problem or have you done anything differently?

@calvinbui thanks for the video! The content is blurred though so I can't really tell what's happening.

[calvinbui]

the video was showing all the new services being created.

as willhughes-au mentioned, if you deploy it w/o remoteDiscoverySelector, it will link every service if the namespace exists

[mateiidavid]

Ahhh @calvinbui I see, that makes sense. Based on the repro instructions I understood that simply re-linking means all services will be mirrored. I have set remote-discovery-selector to an empty string and I can reproduce:

spec:
  clusterCredentialsSecret: cluster-credentials-target
  gatewayAddress: 192.168.224.2
  gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
  gatewayPort: "4143"
  probeSpec:
    path: /ready
    period: 3s
    port: "4191"
  remoteDiscoverySelector: {}
  selector:
    matchLabels:
      mirror.linkerd.io/mirror-to-clusterA: "true"
  targetClusterDomain: cluster.local
  targetClusterLinkerdNamespace: linkerd
  targetClusterName: target

:; k get svc
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes          ClusterIP   10.43.0.1       <none>        443/TCP   23h
podinfo-target      ClusterIP   10.43.117.240   <none>        80/TCP    22h
hokay               ClusterIP   10.43.129.245   <none>        80/TCP    3h9m
kubernetes-target   ClusterIP   10.43.115.138   <none>        443/TCP   7s
nginx-svc-target    ClusterIP   10.43.128.4     <none>        80/TCP    1s