Enhance subctl to run a limited set of connectivity tests on selected nodes

[sridhargaddam]

Currently, subctl verify --only connectivity ... tests validate various use-cases and it usually takes sometime for the tests to complete. While troubleshooting issues, sometimes it would be helpful to have a sub-command which executes a limited number of datapath tests on selected nodes and returns the results. The idea is not to replace the existing e2e tests but to support validating a small sub-set of tests as a quick sanity check when time is a constraint.

The following tests can be validated on the specified nodes:

Vanilla Submariner (i.e., nonGlobalnet) Deployments:

1. Pod to Pod connectivity
2. Pod to Service connectivity

Globalnet Deployments:

1. Pod to Service connectivity

[am-agrawa]

I think it's of great use case to ensure high level connectivity b/w the managed clusters in just a few seconds/minutes of time. Particularly to Regional Disaster Recovery solution of ODF, it would be of great help I believe.

[nirs]

Here what we do in the ramen project as a quick self test to ensure that submariner is working in our minikube testing environment:
https://github.com/RamenDR/ramen/blob/main/test/addons/submariner/test

Basically we follow the usage guide:
https://submariner.io/operations/usage/#2-export-services-across-clusters

0. wait until submariner is ready on both clusters
1. deploy a nginx on one cluster (using nginx slim image)
2. deploy a nettest pod on the other cluster
3. wait until nginx and the nettest pod are ready
4. export the service on cluster where nginx is running
5. import the service on the cluster with the netest pod
6. wait until the export/import services are ready
7. wait until the service DNS name is available (see below)
8. test connectivity with curl, accessing nginx from the other cluster
9. clean up everything

The entire test run in 14 seconds after the initial DNS delay, and this is fast enough
to run it as a self test when creating a test environment, which should happen when a pr
is tested in the CI or for local testing.

Our first issue is to wait until submariner is ready on both clusters we use subctl show all for probing the status, and wait until it returns zero exit code on both clusters:
https://github.com/RamenDR/ramen/blob/cd3cf83fcca08f7f89c8a6445c93d763921dc00d/test/addons/submariner/test#L23

The second issue we have is long delay - up to 60 seconds right after submariner is installed on a new minikube cluster, before the exported/imported service is available on the other cluster. See how we wait for DNS name here:
https://github.com/RamenDR/ramen/blob/cd3cf83fcca08f7f89c8a6445c93d763921dc00d/test/addons/submariner/test#L165

We do this test in both directions - from cluster 1 to cluster 2 and from cluster 2 to cluster 1, by installing the same service with different namespaces, and by interleaving the deploments/waits so we can test both direction at once. This probably can be simplified using threads.
https://github.com/RamenDR/ramen/blob/cd3cf83fcca08f7f89c8a6445c93d763921dc00d/test/addons/submariner/test#L236

It would be nice if subctl would provide a similar test, saving all this effort for every user that want this kind of quick connectivity test.

[tpantelis]

For non-Globalnet connectivity, it runs 12 test specs:

1. [dataplane] Gateway status reporting when a gateway node is configured...
2. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the remote pod is not on a gateway ...
3. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the remote pod is on a gateway ...
4. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is on a gateway and the remote pod is not on a gateway ...
5. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is on a gateway and the remote pod is on a gateway ...
6. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is not on a gateway and the remote service is not on a gateway ...
7. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is not on a gateway and the remote service is on a gateway ...
8. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is on a gateway and the remote service is not on a gateway ...
9. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is on a gateway and the remote service is on a gateway ...
10. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod with HostNetworking connects via TCP to a remote pod when the pod is not on a gateway and the remote pod is not on a gateway ...
11. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod with HostNetworking connects via TCP to a remote pod when the pod is on a gateway and the remote pod is not on a gateway ...
12. [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod in reverse direction when the pod is not on a gateway and the remote pod is not on a gateway ...

@sridhargaddam For a basic verify, I assume we would drop 1, 2 and 10-12? 2-5 are Pod to Pod connectivity tests and 6-9 are Pod to Service connectivity tests, each with varying combinations of gateway and non-gateway nodes. Should we reduce it to just one combination each? eg non-gateway to gateway?

For Globalnet it runs 12 test specs:

1. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway ...
2. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway ...
3. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway ...
4. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway ...
5. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP namespace selector connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway ...
6. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway ...
7. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod matching an egress IP pod selector connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway ...
8. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod with HostNetworking connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway ...
9. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod with HostNetworking connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is not on a gateway ...
10. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote headless service when the pod is not on a gateway and the remote service is not on a gateway ...
11. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote headless service when the pod is on a gateway and the remote service is on a gateway ...
12. [dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service in reverse direction when the pod is not on a gateway and the remote service is not on a gateway ....

[sridhargaddam]

Thanks @tpantelis for summarizing the tests. I think we can run the following tests for basic verify.

non-Globalnet deployments:

[dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is not on a gateway and the remote pod is not on a gateway ...

[dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is on a gateway and the remote pod is on a gateway ...

[dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is not on a gateway and the remote service is not on a gateway ...

[dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is on a gateway and the remote service is on a gateway ...

Globalnet deployments:

[dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is not on a gateway and the remote service is not on a gateway ...

[dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery when a pod connects via TCP to the globalIP of a remote service when the pod is on a gateway and the remote service is on a gateway ...