Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ExemptFeatures knob doesnt work for Experimental Conformance Profile #2513

Closed
arkodg opened this issue Oct 24, 2023 · 2 comments · Fixed by #2515
Closed

ExemptFeatures knob doesnt work for Experimental Conformance Profile #2513

arkodg opened this issue Oct 24, 2023 · 2 comments · Fixed by #2515
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.
Milestone

Comments

@arkodg
Copy link
Contributor

arkodg commented Oct 24, 2023

What happened:

Trying to run all the conformance tests related to Ingress by setting SupportedFeatures: suite.AllFeatures to run all tests and also setting ExemptFeatures: suite.MeshCoreFeatures to disable tests tied to mesh

Sample config

		suite.ExperimentalConformanceOptions{
			Options: suite.Options{
				Client:     mgrClient,
				RestConfig: cfg,
				// This clientset is needed in addition to the client only because
				// controller-runtime client doesn't support non CRUD sub-resources yet (https://github.com/kubernetes-sigs/controller-runtime/issues/452).
				Clientset:            k8sClientset,
				GatewayClassName:     *flags.GatewayClassName,
				Debug:                *flags.ShowDebug,
				CleanupBaseResources: *flags.CleanupBaseResources,
				SupportedFeatures:    suite.AllFeatures,
				ExemptFeatures: suite.MeshCoreFeatures,
			},
			Implementation:      *implementation,
			ConformanceProfiles: conformanceProfiles,
		})

Running with this config causes the mesh tests to run, which is not respecting the ExemptFeatures knob

2023-10-24T00:29:27.1815127Z     --- PASS: TestExperimentalConformance/MeshBasic (0.28s)
2023-10-24T00:29:27.1816429Z         --- PASS: TestExperimentalConformance/MeshBasic/0_request_to_'echo'_should_receive_a_200 (0.27s)
2023-10-24T00:29:27.1817657Z     --- FAIL: TestExperimentalConformance/MeshConsumerRoute (30.13s)
2023-10-24T00:29:27.1819337Z         --- FAIL: TestExperimentalConformance/MeshConsumerRoute/request_from_consumer_route's_namespace_modified_by_HTTPRoute (30.00s)
2023-10-24T00:29:27.1821484Z         --- PASS: TestExperimentalConformance/MeshConsumerRoute/request_not_from_consumer_route's_namespace_not_modified_by_HTTPRoute (0.10s)
2023-10-24T00:29:27.1896361Z     --- FAIL: TestExperimentalConformance/MeshFrontendHostname (30.13s)
2023-10-24T00:29:27.1899950Z         --- FAIL: TestExperimentalConformance/MeshFrontendHostname/Send_to_service_with_wrong_hostname (30.00s)
2023-10-24T00:29:27.1903928Z         --- PASS: TestExperimentalConformance/MeshFrontendHostname/Send_to_other_service_with_matching_hostname (0.10s)
2023-10-24T00:29:27.1905237Z     --- FAIL: TestExperimentalConformance/MeshFrontend (30.15s)
2023-10-24T00:29:27.1940433Z         --- FAIL: TestExperimentalConformance/MeshFrontend/Send_to_service (30.00s)
2023-10-24T00:29:27.1942228Z         --- PASS: TestExperimentalConformance/MeshFrontend/Send_to_pod_IP (0.12s)
2023-10-24T00:29:27.1943985Z     --- FAIL: TestExperimentalConformance/MeshPorts (90.15s)
2023-10-24T00:29:27.1945331Z         --- FAIL: TestExperimentalConformance/MeshPorts/Explicit_port_set,_send_to_that_port (30.00s)
2023-10-24T00:29:27.1946893Z         --- PASS: TestExperimentalConformance/MeshPorts/Explicit_port,_send_to_an_excluded_port (0.10s)
2023-10-24T00:29:27.1948228Z         --- FAIL: TestExperimentalConformance/MeshPorts/No_port_set (30.00s)
2023-10-24T00:29:27.1949408Z         --- FAIL: TestExperimentalConformance/MeshPorts/No_port_set#01 (30.00s)

What you expected to happen:

The exemptFeatures knob should work

How to reproduce it (as minimally and precisely as possible):

Here's a PR that is experiencing the issue envoyproxy/gateway#2056

Anything else we need to know?:

@arkodg arkodg added the kind/bug Categorizes issue or PR as related to a bug. label Oct 24, 2023
@sunjayBhatia
Copy link
Member

related: #2205

arkodg added a commit to arkodg/gateway-api that referenced this issue Oct 24, 2023
@arkodg
Copy link
Contributor Author

arkodg commented Oct 24, 2023

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

3 participants