Tilt up flakes

[fabriziopandini]

What steps did you take and what happened:
Recently when running tilt up, I frequently get

Traceback (most recent call last):
  /Users/fpandini/go/src/sigs.k8s.io/cluster-api/Tiltfile:239:24: in <toplevel>
  /Users/fpandini/go/src/sigs.k8s.io/cluster-api/tilt_modules/cert_manager/Tiltfile:62:10: in deploy_cert_manager
  <builtin>: in local
Error: command ["sh" "-c" "cat << EOF | kubectl apply -f - \napiVersion: v1\nkind: Namespace\nmetadata:\n  name: cert-manager-test\n---\napiVersion: cert-manager.io/v1\nkind: Issuer\nmetadata:\n  name: test-selfsigned\n  namespace: cert-manager-test\nspec:\n  selfSigned: {}\n---\napiVersion: cert-manager.io/v1\nkind: Certificate\nmetadata:\n  name: selfsigned-cert\n  namespace: cert-manager-test\nspec:\n  dnsNames:\n    - example.com\n  secretName: selfsigned-cert-tls\n  issuerRef:\n    name: test-selfsigned\nEOF"] failed.
error: exit status 1
stdout: "namespace/cert-manager-test created\n"
stderr: "Error from server (InternalError): error when creating \"STDIN\": Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": dial tcp 10.96.46.0:443: connect: connection refused\nError from server (InternalError): error when creating \"STDIN\": Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": dial tcp 10.96.46.0:443: connect: connection refused\n"

By running tilt up again then everything works as expected, so this problem is not critical/blocking.

What did you expect to happen:
To tilt up without errors

Anything else you would like to add:
Apparently we are too aggressive in applying cert-manager-test yaml (or probably failing to detect when cert manager is installed and running)

Environment:

• Cluster-api version: main

/kind bug

[fabriziopandini]

/help

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vincepri]

I thought we were waiting for webhooks to be available before starting everything up? Is the command not working anymore?

[fabriziopandini]

The wait is still there, but It seems not working anymore... need to investigate.

[vincepri]

/milestone v0.4

[sbueringer]

@fabriziopandini Does it by any chance only happen with Kubernetes >= 1.21?

[fabriziopandini]

Does it by any chance only happen with Kubernetes >= 1.21?

No, this happens when installing cert-manager on a kind cluster v1.19.1

[CecileRobertMichon]

I'm running into the same thing locally (with kind cluster v1.20.2)

➜  cluster-api git:(new-mp-group) ✗ kind version
kind v0.10.0 go1.16.2 darwin/amd64
➜  cluster-api git:(new-mp-group) ✗ go version
go version go1.16.2 darwin/amd64
➜  cluster-api git:(new-mp-group) ✗ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[vincepri]

https://github.com/kubernetes-sigs/cluster-api/blob/master/tilt_modules/cert_manager/Tiltfile#L56-L58

Given that these are shelling out, it could be the kubectl version used locally? Are your kubectl(s) 1.21? Maybe let's remove the quiet options to see if there is any errors coming from those commands?

[vincepri]

The only change I see w.r.t kubectl wait in 1.21 is kubernetes/kubernetes#97408

[CecileRobertMichon]

I have kubectl v1.19.0 locally (should probably update it...)