Flakes in the machine pools unit tests

[fabriziopandini]

What steps did you take and what happened:
While investigating some unit test failures I some this error happen sometimes:

--- PASS: TestMachinePoolConditions (0.08s)
    --- PASS: TestMachinePoolConditions/all_conditions_true (0.01s)
    --- PASS: TestMachinePoolConditions/boostrap_not_ready (0.02s)
    --- PASS: TestMachinePoolConditions/bootstrap_not_ready_with_fallback_condition (0.02s)
    --- PASS: TestMachinePoolConditions/infrastructure_not_ready (0.00s)
    --- PASS: TestMachinePoolConditions/infrastructure_not_ready_with_fallback_condition (0.03s)
=== RUN   TestAPIs
Running Suite: Controller Suite
===============================
Random Seed: 1610553618
....
••E0113 16:00:39.205134   12454 certwatcher.go:144] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0113 16:00:39.205196   12454 certwatcher.go:149] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-204656340/tls.crt: no such file or directory"  
E0113 16:00:39.205232   12454 certwatcher.go:144] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0113 16:00:39.205261   12454 certwatcher.go:149] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-204656340/tls.crt: no such file or directory"  
I0113 16:00:39.205404   12454 server.go:231] controller-runtime/webhook "msg"="shutting down webhook server"  
STEP: tearing down the test environment
------------------------------
Failure [0.652 seconds]
[AfterSuite] AfterSuite 
/home/prow/go/src/sigs.k8s.io/cluster-api/exp/controllers/suite_test.go:66
  Expected success, but got an error:
      <*errors.StatusError | 0xc000348aa0>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "Timeout: failed waiting for *v1alpha4.MachinePool Informer to sync",
              Reason: "Timeout",
              Details: {Name: "", Group: "", Kind: "", UID: "", Causes: nil, RetryAfterSeconds: 0},
              Code: 504,
          },
      }
      Timeout: failed waiting for *v1alpha4.MachinePool Informer to sync
  /home/prow/go/src/sigs.k8s.io/cluster-api/exp/controllers/suite_test.go:60
------------------------------
Ran 9 of 9 Specs in 20.685 seconds
FAIL! -- 9 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (20.69s)
FAIL
FAIL	sigs.k8s.io/cluster-api/exp/controllers	21.138s

The error seems to be in the tear-down sequence, not in the test itself 🤔

Environment:

• Cluster-api version: Main

/kind bug

[fabriziopandini]

/area testing
/milestone v0.4.x

[k8s-ci-robot]

@fabriziopandini: The provided milestone is not valid for this repository. Milestones in this repository: [Next, v0.3.x, v0.4.0]

Use /milestone clear to clear the milestone.

In response to this:

/area testing
/milestone v0.4.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

@CecileRobertMichon @detiber @vincepri I could use some help here.

"Timeout: failed waiting for *v1alpha4.MachinePool Informer to sync" happens only sometimes.

MachinePool CRDs is registered in testenv together with the other CAPI CRDs, so the fact that this error only happens for this kind is driving me to assume the problem is in exp/api/v1alpha4.

However, the only thing a little bit odd that I can find in the MachinePool types is t that they are disabling conversions, but I don't have the full context here

cluster-api/exp/api/v1alpha4/machinepool_types.go

Line 208 in 207aadd

// +k8s:conversion-gen=false

Does anything above triggers some bell?
Do you think I'm going in the right direction while investigating this issue?

[vincepri]

Is the error coming from the StartManager function? From above it seems like it, but it also seems tests did run 🤔

[fabriziopandini]

@vincepri, yes the error is from StartManager, and the test is run

My current assumption is that the test is too fast, and test env is stopped even before StartManager is completed (which is something that can happen given that most of the initialization stuff happens in goroutines, so it is not blocking). However, I don't know yet how can we tackle this...

[faiq]

/reopen

[k8s-ci-robot]

@faiq: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[faiq]

Hey folks, just wanted to register that I'm currently experiencing this problem as well. To me it seems like this is occurring because the files are never installed.

E0401 12:45:18.942558  504688 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0401 12:45:18.943325  504688 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-767561487/tls.crt: no such file or directory"  
I0401 12:45:18.942641  504688 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="clusterresourcesetbinding"

I think this can be solved by using this method to write the files to disk
https://github.com/kubernetes-sigs/controller-runtime/blob/10ae090c1d3ac0c560dfa1a29b2517eb8d74442b/pkg/envtest/webhook.go#L163

in this method in the envtest helper.

cluster-api/test/helpers/envtest.go

Line 222 in 2ea6cfb

func initializeWebhookInEnvironment() {

[fabriziopandini]

@faiq do you mind to move your comment to a new issue, so we can track this problem without being confused by the previous discussion thread