HA: With "cluster to cluster heartbeats" to identify the cluster down and update the lighthouse DNS cache as soon as possible

[ming-ddtechcg]

I am running v0.6.1 on the two Openshift 4.4.20 clusters with "--cable-driver libreswan" while they joined together, say "cluster A" and "cluster B".

Deployed the test nginx-unprivileged case from "KIND (Local Environment)" to both clusters and expose services with kubectl and subctl. Performed nslookup with the FQDN service name in the pod "tmp-shell", and two service IP addresses are returned as round-robin.

The test scenario is that to shut down the cluster A (I bring down all worker nodes) and the service will be routed to the cluster B only:

the result of the testing to return the cluster B service IP took approximately 25 minutes (the timings were more or less sometimes).

There was the cluster B Gateway describe while the cluster A was down after 10 minutes:

$ kubectl describe Gateway -n submariner-operator  
Name:         clusterb-62kfm-worker-0-2pvhw
Namespace:    submariner-operator
Labels:       <none>
Annotations:  update-timestamp: 1600352166
API Version:  submariner.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2020-09-17T13:06:49Z
  Generation:          2
  Resource Version:    5746231
  Self Link:           /apis/submariner.io/v1/namespaces/submariner-operator/gateways/clusterb-62kfm-worker-0-2pvhw
  UID:                 0886e928-8706-4103-a3cb-5cf1a8390e51
Status:
  Connections:
    Endpoint:
      Backend:      libreswan
      cable_name:   submariner-cable-cluster-clustera-10-5-33-224
      cluster_id:   cluster-a
      Hostname:     worker0.clustera.test.com
      nat_enabled:  false
      private_ip:   10.5.33.224
      public_ip:    
      Subnets:
        172.30.0.0/16
        10.128.0.0/16
    Status:          connected
    Status Message:  
  Ha Status:         active
  Local Endpoint:
    Backend:      libreswan
    cable_name:   submariner-cable-cluster-b-10-5-71-225
    cluster_id:   cluster-c16
    Hostname:     clusterb-62kfm-worker-0-2pvhw
    nat_enabled:  false
    private_ip:   10.5.71.225
    public_ip:    
    Subnets:
      172.31.0.0/16
      10.132.0.0/14
  Status Failure:  
  Version:         v0.6.0
Events:            <none>

Recommendation:

If the lighthouse DNS can remove the down cluster service IP address off in 5 to 10 seconds (assume the good networking environment).

Or, we leave the options to users to configure it:

To define the parameter (--ha-heartbeat-interval x --ha-heartbeat-failure-count y) to declare the heartbeat ping failure between clusters while performing the subctl join. Like:

--ha-heartbeat-interval
it is a timer for how frequent sending the heartbeat from a health cluster to other joined clusters in x seconds. the range is 1 second <= x <= 600 seconds (default is 30 seconds)

--ha-heartbeat-failure-count
the number of the consecutive failure to declare the losing connection from the specified cluster 3 <= y <=50 (default is 5)

[sridhargaddam]

Thanks for reporting the issue @ming-ddtechcg

[mangelajo]

I've added some more labels, I think this should also be considered as bug

[nyechiel]

@aswinsuryan before we move this to done, what's the test coverage for this? Is it part of our E2E? Also, what about docs?

[aswinsuryan]

@aswinsuryan before we move this to done, what's the test coverage for this? Is it part of our E2E? Also, what about docs?

Oh it was auto-closed with API patch merge. The implementation is yet to be added and UT and E2E. I will raise two separate issues for doc and e2e.

[mangelajo]

@aswinsuryan Isn't this ready in 0.8-rc1 already?

[mangelajo]

ah, just waiting for verification?

[nyechiel]

ah, just waiting for verification?

Yes, we want @manosnoam to review and verify.

[manosnoam]

@nyechiel @aswinsuryan
I've opened E2E test request for this feature: #1041