Deploy chart with more than one replica fail webhooks

[ghost]

Bug description

When I deploy the chart with more than one replica, the webhooks only start listenting in the leader. This causes that when requests are started and reach the replica, it returns a 500 error connection refused because the webhooks have not been deployed.

The reason is because the chart deployment create an empty secrets (ca-secret and tls-secret) and the manager container that has been set leader, in first init, detect the empty secrets, creates certificates and fill secrets with them. After this process, the manager container will be restarted automatically but the replicas not, that causes the webhooks never start in replicas until deployment/pod restart and detect secrets with certificates filled.

I tested fill the secrets in chart before install it and it works fine because all replicas start with the secrets filled.

How to reproduce

Steps to reproduce the behavior:

Set default chart values and change replicaCount: to more than one replica.

Expected behavior

The webhooks will be start in all replicas for answer the requests in HA.

Logs

Logs of manager container that has been elected leader in first execution:

[bash]$ k logs --previous capsule-controller-manager-946d5bb5f-4xfbn -c manager -n capsule-system
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Capsule Version  "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Build from: "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Build date: "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Go Version: go1.15.11"}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":"2021-08-27T10:34:13.077Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2021-08-27T10:34:13.081Z","logger":"setup","msg":"starting with following options:","metricsAddr":"127.0.0.1:8080","enableLeaderElection":true,"forceTenantPrefix":true}
{"level":"info","ts":"2021-08-27T10:34:13.082Z","logger":"setup","msg":"starting manager"}
I0827 10:34:13.082581       1 leaderelection.go:243] attempting to acquire leader lease  capsule-system/42c733ea.clastix.capsule.io...
{"level":"info","ts":"2021-08-27T10:34:13.082Z","logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
W0827 10:34:13.088242       1 warnings.go:67] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0827 10:34:13.089615       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0827 10:34:13.092510       1 warnings.go:67] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0827 10:34:13.092677       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
I0827 10:34:13.115634       1 leaderelection.go:253] successfully acquired lease capsule-system/42c733ea.clastix.capsule.io
{"level":"debug","ts":"2021-08-27T10:34:13.115Z","logger":"controller-runtime.manager.events","msg":"Normal","object":{"kind":"ConfigMap","namespace":"capsule-system","name":"42c733ea.clastix.capsule.io","uid":"022df253-0d34-480b-b38d-a2e8f310dea4","apiVersion":"v1","resourceVersion":"1636"},"reason":"LeaderElection","message":"capsule-controller-manager-946d5bb5f-4xfbn_602c0cfc-24a5-410e-b202-3d844c800814 became leader"}
{"level":"debug","ts":"2021-08-27T10:34:13.115Z","logger":"controller-runtime.manager.events","msg":"Normal","object":{"kind":"Lease","namespace":"capsule-system","name":"42c733ea.clastix.capsule.io","uid":"5844e58d-9854-4f0b-88a5-3b8cf38e3201","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1637"},"reason":"LeaderElection","message":"capsule-controller-manager-946d5bb5f-4xfbn_602c0cfc-24a5-410e-b202-3d844c800814 became leader"}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controllers.Rbac","msg":"setting up ClusterRoles","ClusterRole":"capsule-namespace-provisioner"}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.service","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Service","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.secret","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Secret","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.endpoints","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Endpoints","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.secret","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Secret","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.clusterrole","msg":"Starting EventSource","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRole","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.clusterrolebinding","msg":"Starting EventSource","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRoleBinding","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.endpointslice","msg":"Starting EventSource","reconcilerGroup":"discovery.k8s.io","reconcilerKind":"EndpointSlice","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.183Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.283Z","logger":"controller.endpointslice","msg":"Starting Controller","reconcilerGroup":"discovery.k8s.io","reconcilerKind":"EndpointSlice"}
{"level":"info","ts":"2021-08-27T10:34:13.283Z","logger":"controller.endpointslice","msg":"Starting workers","reconcilerGroup":"discovery.k8s.io","reconcilerKind":"EndpointSlice","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.283Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.clusterrole","msg":"Starting Controller","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRole"}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.endpoints","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Endpoints"}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.secret","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Secret"}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.service","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Service"}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.clusterrolebinding","msg":"Starting Controller","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRoleBinding"}
{"level":"info","ts":"2021-08-27T10:34:13.284Z","logger":"controller.secret","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Secret"}
{"level":"info","ts":"2021-08-27T10:34:13.290Z","logger":"controllers.Rbac","msg":"setting up ClusterRoles","ClusterRole":"capsule-namespace-deleter"}
{"level":"info","ts":"2021-08-27T10:34:13.294Z","logger":"controllers.Rbac","msg":"setting up ClusterRoleBindings","ClusterRoleBinding":"capsule-namespace-provisioner"}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.endpoints","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Endpoints","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.secret","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Secret","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.service","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Service","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.clusterrole","msg":"Starting workers","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRole","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.clusterrolebinding","msg":"Starting workers","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRoleBinding","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.384Z","logger":"controller.secret","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Secret","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.385Z","logger":"controllers.CA","msg":"Reconciling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:13.390Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.415Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.435Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.476Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.485Z","logger":"controller.tenant","msg":"Starting EventSource","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-27T10:34:13.557Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:13.615Z","logger":"controller.tenant","msg":"Starting Controller","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant"}
{"level":"info","ts":"2021-08-27T10:34:13.615Z","logger":"controller.tenant","msg":"Starting workers","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant","worker count":1}
{"level":"info","ts":"2021-08-27T10:34:13.718Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:14.039Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:14.693Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:15.995Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:18.556Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:23.715Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:33.028Z","logger":"controllers.CA","msg":"Handling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.028Z","logger":"controllers.CA","msg":"Updating CA secret with new PEM and RSA","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.346Z","logger":"controllers.CA","msg":"Capsule CA has been updated, we need to trigger TLS update too","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.347Z","logger":"controllers.CA","msg":"Reconciliation completed, processing back in 87647h59m40s","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.347Z","logger":"controllers.CA","msg":"Reconciling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.347Z","logger":"controllers.CA","msg":"Handling CA Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.347Z","logger":"controllers.CA","msg":"Updating CA secret with new PEM and RSA","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.549Z","logger":"controllers.CA","msg":"Reconciliation completed, processing back in 87647h59m40s","Request.Namespace":"capsule-system","Request.Name":"capsule-ca"}
{"level":"info","ts":"2021-08-27T10:34:33.956Z","logger":"controllers.Tls","msg":"Reconciling TLS Secret","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:33.956Z","logger":"controllers.Tls","msg":"Missing Capsule TLS certificate","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controllers.Tls","msg":"Capsule TLS certificates has been updated, we need to restart the Controller","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controllers.Tls","msg":"Reconciliation completed, processing back in 4320h0m0s","Request.Namespace":"capsule-system","Request.Name":"capsule-tls"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.tenant","msg":"Stopping workers","reconcilerGroup":"capsule.clastix.io","reconcilerKind":"Tenant"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.secret","msg":"Stopping workers","reconcilerGroup":"","reconcilerKind":"Secret"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.secret","msg":"Stopping workers","reconcilerGroup":"","reconcilerKind":"Secret"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.endpoints","msg":"Stopping workers","reconcilerGroup":"","reconcilerKind":"Endpoints"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.clusterrolebinding","msg":"Stopping workers","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRoleBinding"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.clusterrole","msg":"Stopping workers","reconcilerGroup":"rbac.authorization.k8s.io","reconcilerKind":"ClusterRole"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.service","msg":"Stopping workers","reconcilerGroup":"","reconcilerKind":"Service"}
{"level":"info","ts":"2021-08-27T10:34:50.335Z","logger":"controller.endpointslice","msg":"Stopping workers","reconcilerGroup":"discovery.k8s.io","reconcilerKind":"EndpointSlice"}

As you can see, fill the secrets and reboot.
Logs of manager container after auto reboot:

[bash]# k logs capsule-controller-manager-946d5bb5f-4xfbn -c manager -n capsule-system
{"level":"info","ts":"2021-08-27T10:34:50.727Z","logger":"setup","msg":"Capsule Version  "}
{"level":"info","ts":"2021-08-27T10:34:50.727Z","logger":"setup","msg":"Build from: "}
{"level":"info","ts":"2021-08-27T10:34:50.727Z","logger":"setup","msg":"Build date: "}
{"level":"info","ts":"2021-08-27T10:34:50.727Z","logger":"setup","msg":"Go Version: go1.15.11"}
{"level":"info","ts":"2021-08-27T10:34:50.727Z","logger":"setup","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":"2021-08-27T10:34:51.394Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2021-08-27T10:34:51.421Z","logger":"setup","msg":"starting with following options:","metricsAddr":"127.0.0.1:8080","enableLeaderElection":true,"forceTenantPrefix":true}
{"level":"info","ts":"2021-08-27T10:34:51.421Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-ingress"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-v1-pvc"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-v1-registry"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-external-service-ips"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-namespace-owner-reference"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-v1-namespace-quota"}
{"level":"info","ts":"2021-08-27T10:34:51.422Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-v1-network-policy"}
{"level":"info","ts":"2021-08-27T10:34:51.423Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-v1-namespace-tenant-prefix"}
{"level":"info","ts":"2021-08-27T10:34:51.423Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validating-v1-tenant"}
{"level":"info","ts":"2021-08-27T10:34:51.424Z","logger":"setup","msg":"starting manager"}

After rebooting, you can see how the webhooks start because it has detected the full secrets.
Logs of manager container that is replica:

[bash]# k logs capsule-controller-manager-946d5bb5f-rpskn -c manager -n capsule-system
{"level":"info","ts":"2021-08-27T10:34:12.421Z","logger":"setup","msg":"Capsule Version  "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Build from: "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Build date: "}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Go Version: go1.15.11"}
{"level":"info","ts":"2021-08-27T10:34:12.422Z","logger":"setup","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":"2021-08-27T10:34:13.076Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":"2021-08-27T10:34:13.078Z","logger":"setup","msg":"starting with following options:","metricsAddr":"127.0.0.1:8080","enableLeaderElection":true,"forceTenantPrefix":true}
{"level":"info","ts":"2021-08-27T10:34:13.078Z","logger":"setup","msg":"starting manager"}

As you can see, the webhooks not start at any time, even if secrets have been filled out. It also needs to be restarted in order to start the webhooks

Additional context

• Capsule version: 0.0.18
• Helm Chart version: 0.0.4
• Kubernetes version: 1.20.7

[prometherion]

Hey @adrianhernandez-stratio, thanks for the useful bug report: it could sound silly but speaking for myself I never tested the HA from the scratch installation, that's awkward! 😆

I guess we can nail this issue easily in two ways:

1. instead of letting Capsule kill itself, we could perform a Deployment rollout, deleting all the Pods with the given selector.
2. using the Helm hook to force the Deployment start in a single replica in case of missing Secret, and then scaling back to the desired value.

Honestly, the second option isn't portable for the single YAML manifest and seems to break the immutability of infrastructure as code, although speaking of Pods.

If you have any suggestions I would be glad to hear those, as well as contributions!

[ghost]

Hi @prometherion, thanks you for quick response.

I think that the best option that you propose is the first one, as you well indicate. It can be confusing to deploy the chart with one replica and then scale.

Cheers!