add readiness probe #84
Conversation
Currently, manager pods turn Ready as soon as their binary starts. However, it takes ~40 more seconds for the webhook to actually start serving requests. In order to avoid that, we introduce a readiness probe that marks the pod as Ready only once the server starts. Please note that there is no way to do it properly with the current controller-runtime. We don't have a hook point that would get executed once the webhook has started. Therefore we have a Sleep there. That should be dropped once kubernetes-sigs/cluster-api#1855 is implemented. Signed-off-by: Petr Horacek <[email protected]>
| command: | ||
| - cat | ||
| - /tmp/ready | ||
| initialDelaySeconds: 5 |
There was a problem hiding this comment.
why do we need any golang code for this hack? Running cat of an ever-existing file every 5 seconds feels silly.
Would it not be enough to have
initialDelaySeconds: 20
with a fat comment carrying the same explanation?
There was a problem hiding this comment.
We need to create the file once the server starts.
I don't think saving 3 cat runs is worth an extra line and comment.
We can replace cat with HTTP GET once controller-runtime implements support for it. I'm not aware of any option that would allow us to stop querying once it succeeds once. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
There was a problem hiding this comment.
I was speaking about something like
readinessProbe:
exec:
command:
- true
initialDelaySeconds: 20
periodSeconds: 99999
and dropping the golang changes altogether. But I'm splitting hairs here. The important thing is that you have found the races that @AlonaKaplan has spoken about.
There was a problem hiding this comment.
That would not help. True can succeed as soon as the pod starts. That would be the same as happens with default readiness.
The point here is, that we write the file only once we go through all the initialization and we start the webhook server. In other words, we are marked Ready only once we are able to respond to HTTPS requests for /mutate-virtualmachines.
The extra 20 seconds wait is on top of all the initialization (collecting existing MACs, getting certificates). Those 20 seconds are waiting until the Start hopefully really starts serving.
There was a problem hiding this comment.
We still have races in the test code #83. Unfortunately, the tests are more resilient than the kubemacpool code, so it is all hidden.
In the current code, setRange and deleteLeaderManager functions are not waiting for the old pods to disappear and new ones to become available. Due to that, tests using this functions don't have any webhook available or worse, they are being served by the old one. This issue was not exposed due to Eventually we have around all the Create calls. This commit makes sure the old pods are removed but since the readiness probe proposed in k8snetworkplumbingwg#84 is not merged yet, a sleep is used here. Signed-off-by: Petr Horacek <[email protected]> Signed-off-by: Alona Kaplan <[email protected]>
In the current code, setRange and deleteLeaderManager functions are not waiting for the old pods to disappear and new ones to become available. Due to that, tests using this functions don't have any webhook available or worse, they are being served by the old one. This issue was not exposed due to Eventually we have around all the Create calls. This commit makes sure the old pods are removed but since the readiness probe proposed in k8snetworkplumbingwg#84 is not merged yet, a sleep is used here. Signed-off-by: Petr Horacek <[email protected]> Signed-off-by: Alona Kaplan <[email protected]>
Currently, manager pods turn Ready as soon as their binary starts.
However, it takes ~40 more seconds for the webhook to actually start
serving requests. In order to avoid that, we introduce a readiness
probe that marks the pod as Ready only once the server starts.
Please note that there is no way to do it properly with the current
controller-runtime. We don't have a hook point that would get executed
once the webhook has started. Therefore we have a Sleep there. That
should be dropped once
kubernetes-sigs/cluster-api#1855
is implemented.