CNCF conformance pass for AWS CAPI clusters

[alex-dabija]

Run CNCF conformance tests for CAPA clusters. Issues identified during testing need to be addressed in order to have the test suite pass.

[njuettner]

We need to add this label on the master node, otherwise cluster-autoscaler can't be scheduled:
kubernetes.io/role: master

[njuettner]

Status: failed
Total: 5484
Passed: 300
Failed: 5
Skipped: 5179

Failed tests:
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance]
[sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to create a functioning NodePort service [Conformance]
[sig-network] Services should have session affinity timeout work for NodePort service [LinuxOnly] [Conformance]

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

[njuettner]

I just ran the tests again, multiple time. Same output 😞

$ sonobuoy results 202109300943_sonobuoy_a72ab36f-463d-4897-832e-5276a886a8c3.tar.gz
Plugin: e2e
Status: failed
Total: 5484
Passed: 300
Failed: 5
Skipped: 5179

Failed tests:
[sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to create a functioning NodePort service [Conformance]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance]
[sig-network] Services should have session affinity timeout work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]

Plugin: systemd-logs
Status: passed
Total: 4
Passed: 4
Failed: 0
Skipped: 0

[njuettner]

From upstream:

This is a result of #2798 . The conformance tests want to use node ports that aren't open by default in the security group rules.

However, we have to find a balance - most customers/users I speak to currently balk at the existing open ports, which is why #2798 allows for its customisation. You can however make it work, by adjusting "cniIngressRules" and adding the used ports there.

Set CNIIngressRules on the awsCluster resource with the rules required. https://cluster-api-aws.sigs.k8s.io/crd/index.html#infrastructure.cluster.x-k8s.io/v1alpha4.CNIIngressRules

These were introduced in v1alpha3, so you don't need #2798 .

I'm not sure exactly what you'll need - I think you need to add UDP 30000-65535, best to check the source code in the main kubernetes repo for what's required.

---

After digging a bit deeper we found the issue, CNIIngressRules are already set and wasn't the issue. The issue is when creating CAPA cluster we do not explicitly set --node-ip. This leads to adding IP's from AWS CNI which is wrong. Sonobuoy shouldn't use those IPs to test services with nodeport.

After setting the flag all tests finally passed. I'll create a follow-up issue.