Cumbersome experimental conformance reporting suite UX

[sunjayBhatia]

Currently in Contour we use these conformance suite options to ensure we run all tests by default, but intentionally have to exclude a few that either don't apply or don't pass. In our case, we exclude the Mesh feature and one of the extended support HTTPProxy tests.

Doing this in the experimental conformance suite currently is not as simple as that to get the full report on supported extended features out. Instead you need something like:

	cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
		Options: suite.Options{
                        ...
                        // Note this vs. using the EnableAllSupportedFeatures and ExemptFeatures field
			SupportedFeatures: suite.AllFeatures.Delete(suite.MeshCoreFeatures.UnsortedList()...),
			SkipTests: []string{
				tests.HTTPRouteRedirectPortAndScheme.ShortName,
			},
		},
		Implementation: conformance_v1alpha1.Implementation{
                         ...
		},
		ConformanceProfiles: sets.New(
			suite.HTTPConformanceProfileName,
			suite.TLSConformanceProfileName,
		),
	})

It would be great if using the same suite.Options setup was possible between the two versions, especially if the current one is eventually replaced by the version that can report conformance results.

Notes:

• Logic for building up the experimental suite features is here:

  gateway-api/conformance/utils/suite/experimental_suite.go

  Lines 114 to 147 in 6d0e6cc

  	if s.EnableAllSupportedFeatures {
  	s.SupportedFeatures = AllFeatures
  	} else {
  	if s.SupportedFeatures == nil {
  	s.SupportedFeatures = sets.New[SupportedFeature]()
  	}
  	for _, conformanceProfileName := range s.ConformanceProfiles.UnsortedList() {
  	conformanceProfile, err := getConformanceProfileForName(conformanceProfileName)
  	if err != nil {
  	return nil, fmt.Errorf("failed to retrieve conformance profile: %w", err)
  	}
  	// the use of a conformance profile implicitly enables any features of
  	// that profile which are supported at a Core level of support.
  	for _, f := range conformanceProfile.CoreFeatures.UnsortedList() {
  	if !s.SupportedFeatures.Has(f) {
  	s.SupportedFeatures.Insert(f)
  	}
  	}
  	for _, f := range conformanceProfile.ExtendedFeatures.UnsortedList() {
  	if s.SupportedFeatures.Has(f) {
  	if suite.extendedSupportedFeatures[conformanceProfileName] == nil {
  	suite.extendedSupportedFeatures[conformanceProfileName] = sets.New[SupportedFeature]()
  	}
  	suite.extendedSupportedFeatures[conformanceProfileName].Insert(f)
  	} else {
  	if suite.extendedUnsupportedFeatures[conformanceProfileName] == nil {
  	suite.extendedUnsupportedFeatures[conformanceProfileName] = sets.New[SupportedFeature]()
  	}
  	suite.extendedUnsupportedFeatures[conformanceProfileName].Insert(f)
  	}
  	}
  	}
  	}

• vs. standard logic here:

  gateway-api/conformance/utils/suite/suite.go

  Lines 92 to 105 in 6d0e6cc

  	switch {
  	case s.EnableAllSupportedFeatures == true:
  	s.SupportedFeatures = AllFeatures
  	case s.SupportedFeatures == nil:
  	s.SupportedFeatures = StandardCoreFeatures
  	default:
  	for feature := range StandardCoreFeatures {
  	s.SupportedFeatures.Insert(feature)
  	}
  	}
  	for feature := range s.ExemptFeatures {
  	s.SupportedFeatures.Delete(feature)
  	}

• Seems like we can simplify the experimental suite logic a bit and add in support for the exempt features etc.

[shaneutt]

Seems like a good area for improvement, thank you for trying out the experimental conformance profile test suite and providing some feedback!

/triage accepted

I don't think we need to necessarily worry about this for GA however, so I'm not intending to add it to that milestone for now but instead as something to follow up on post-GA:

/priority important-longterm

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

/cc @mlavacca

[mlavacca]

Thanks @sunjayBhatia, this would look like a good improvement in usability

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

+1, even if we should be careful not to lose track of this one, even because when the experimental will become the new standard, most likely we'll be deleting the standard one (or deprecating it), and we should take care of adding this improvement there.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten