Make subctl verify output more useful

[mkolesnik]

What would you like to be added:
As a developer, and as an end user, I'd like to know which test is being verified and have a clearer print out which is more in line with how other subctl commands behave (e.g join, diagnose, etc).

e.g.

$ subctl verify --only connectivity --kubecontexts "$(generate_kubecontexts)"
Verifying connectivity
   ✓ [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote pod when the pod is on a gateway and the remote pod is on a gateway
   ✓ [dataplane] Basic TCP connectivity tests across clusters without discovery when a pod connects via TCP to a remote service when the pod is not on a gateway and the remote service is on a gateway
   ...

For printing more detailed information we have a --verbose flag which can print out the test specifics for each test

Why is this needed:
As an end user and as a developer, the output from subctl verify is quite useless right now:

$ subctl verify --only connectivity --kubecontexts "$(generate_kubecontexts)"
Performing the following verifications: connectivity
Running Suite: Submariner E2E suite
===================================
Random Seed: 1656333811
Will run 22 of 41 specs
•SSSSSSSSSS••••••••••SSSSSS
------------------------------
S [SKIPPING] [0.091 seconds]
[dataplane-globalnet] Basic TCP connectivity tests across overlapping clusters without discovery
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_gn_pod_connectivity.go:28
  when a pod connects via TCP to the globalIP of a remote service
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_gn_pod_connectivity.go:58
    when the pod is not on a gateway and the remote service is not on a gateway
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_gn_pod_connectivity.go:65
should have sent the expected data from the pod to the other pod [It]
github.com/submariner-io/[email protected]/test/e2e/dataplane/tcp_gn_pod_connectivity.go:35
Jun 27 12:45:14.400: Globalnet is not enabled, skipping the test...
      github.com/submariner-io/[email protected]/test/e2e/framework/logging.go:60
------------------------------
              < 10 more skipped test outputs here >
------------------------------
SSS
Ran 11 of 41 Specs in 103.718 seconds
SUCCESS! -- 11 Passed | 0 Failed | 0 Pending | 30 Skipped

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[dfarrell07]

Talking about this on the scrub, this still seems useful. Might also be good to expose a more clean way to run specific tests once verify shows the names of the tests (not just test suites as we do now). Would help with debugging, to have fewer logs.

[tpantelis]

Unfortunately, ginkgo doesn't allow a custom Reporter instance to be hooked in so we have little flexibility wrt the output. We also have no visibility into which specs it will/did run. In order for it to output the names of the specs that it runs, we have to set the Verbose flag in the ReporterConfig, which is what we do when the --verbose arg is passed. But this also outputs the steps produced via the ginkgo.By function. What we can do is to always enable Verbose and have --verbose control the By output. shipyard already provides an override for ginkgo.By which is used in some places - we would need to change all occurrences of ginkgo.By to framework.By and set framework.By appropriately.

Here's what the output looks like currently w/o specifying --verbose (running just the single compliance test for simplicity):

Performing the following verifications: compliance
Running Suite: Submariner E2E suite - /home/pantelis/go/src/github.com/submariner-io/subctl
===========================================================================================
Random Seed: 1

Will run 1 of 24 specs
SSSSSSSSSSSSSSS•SSSSSSSS

Ran 1 of 24 Specs in 0.032 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 23 Skipped

Here's what it looks like with the proposed changes with Verbose set w/o --verbose and the By output:

Performing the following verifications: compliance
Running Suite: Submariner E2E suite - /home/pantelis/go/src/github.com/submariner-io/subctl
===========================================================================================
Random Seed: 1

Will run 1 of 24 specs
------------------------------
[SynchronizedBeforeSuite] 
github.com/submariner-io/[email protected]/test/e2e/e2e.go:40
[SynchronizedBeforeSuite] PASSED [0.023 seconds]
------------------------------
SSSSSSSSSSSSSSS
------------------------------
FIPS when FIPS mode is enabled for the active gateway node should use FIPS mode in the libreswan cable driver [compliance]
github.com/submariner-io/[email protected]/test/e2e/compliance/fips.go:36
• [0.006 seconds]
------------------------------
SSSSSSSS
------------------------------
[SynchronizedAfterSuite] 
github.com/submariner-io/[email protected]/test/e2e/e2e.go:53
[SynchronizedAfterSuite] PASSED [0.000 seconds]
------------------------------

Ran 1 of 24 Specs in 0.030 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 23 Skipped

We do get extra output for SynchronizedBeforeSuite/SynchronizedAfterSuite - we can eliminate that by calling framework.BeforeSuite/framework.RunCleanupActions directly. The code location of the spec is also printed out but there's nothing we can do about that.

Here's the full output with --verbose:

Performing the following verifications: compliance
Running Suite: Submariner E2E suite - /home/pantelis/go/src/github.com/submariner-io/subctl
===========================================================================================
Random Seed: 1

Will run 1 of 24 specs
------------------------------
[SynchronizedBeforeSuite] 
github.com/submariner-io/[email protected]/test/e2e/e2e.go:40
  STEP: Creating kubernetes clients @ 03/29/24 11:17:51.554
  STEP: Creating submariner clients @ 03/29/24 11:17:51.558
  STEP: Creating lighthouse clients @ 03/29/24 11:17:51.56
[SynchronizedBeforeSuite] PASSED [0.021 seconds]
------------------------------
SSSSSSSSSSSSSSS
------------------------------
FIPS when FIPS mode is enabled for the active gateway node should use FIPS mode in the libreswan cable driver [compliance]
github.com/submariner-io/[email protected]/test/e2e/compliance/fips.go:36
  STEP: Creating namespace objects with basename "fips-gateway-status" @ 03/29/24 11:17:51.575
  STEP: Generated namespace "e2e-tests-fips-gateway-status-q7bqb" in cluster "cluster1" to execute the tests in @ 03/29/24 11:17:51.577
  STEP: Find a cluster with FIPS enabled
   @ 03/29/24 11:17:51.577
  STEP: Deleting namespace "e2e-tests-fips-gateway-status-q7bqb" on cluster "cluster1" @ 03/29/24 11:17:51.578
• [0.007 seconds]
------------------------------
SSSSSSSS
------------------------------
[SynchronizedAfterSuite] 
github.com/submariner-io/[email protected]/test/e2e/e2e.go:53
[SynchronizedAfterSuite] PASSED [0.000 seconds]
------------------------------

Ran 1 of 24 Specs in 0.028 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 23 Skipped