CI: How to run tests with and without Feature Gates

[markmandel]

Context: #1397 (comment)

The concern is:

Is the idea that we turn on all feature gates for develpers? Does that make it so that the developer testing experience diverges too much from the production experience?

How do we make sure that alpha and beta features work while enabled, but also don't break stable features when disabled. This is primarily a concern with e2e tests.

The flip side concern is that a matrix of feature gate testing seems extremely complicated and time consuming.

[markmandel]

How does Kubernetes solve this? There must be prior art, I would think?

[roberthbailey]

Kubernetes solves this at least in part by running a huge test matrix.

Some examples:

• GCE Tests have different tests for different configurations, such as gce-alpha-api, gce-ha-master, gce-multizone, ip-alias, etc. And then a bunch of those test suites are repeated in the https://k8s-testgrid.appspot.com/google-gci section to test a different node image type.
• GKE Tests cover a whole bunch of cluster configuration options of GKE clusters, including things like gke-canary, gci-gke-alpha-features, and a bunch of tests that appear to be pinned to release branches.
• Conformance Tests which runs the same suite of tests against a whole bunch of different environments / providers to ensure platform consistency.

This is all built on top of prow + testgrid, which makes it easy to spin up new test suites. But if you aren't careful, it can also become costly too as you expand the matrix to cover such a large variety of tests.

[markmandel]

Eep. Sounds scary.

I wonder if we could suffice with 2 e2e tests - one with all features enabled, and one without any (and skipping any tests wherein there are feature flags enabled). Would that be sufficient for the moment?

Might want to run them in parallel on separate clusters?

[markmandel]

This might also be the impetus to finally do the work to move our testing infra off Cloud Builder -- we may have met it's limits at this stage.

[aLekSer]

We need to be able to filter out new E2E tests for example : TestGameServerWithPortsMappedToMultipleContainers when we deploy Agones without FeatureGates.
We probably can separate alpha tests with build tags
https://mickey.dev/posts/go-build-tags-testing/
I think @akremsa can start doing this separation of tests

[markmandel]

Build tags make sense to me 👍 sounds like a good first approach.

[roberthbailey]

Can we mark this as completed? We've had 2 e2e tests (with and without feature gates) for quite some time now and it seems to be providing pretty good test coverage.

[markmandel]

I think we can!

Let's mark as complete and if we ever want to expand coverage we can open a new ticket.