Store: make the liveness probe timeout configurable #286
Conversation
Under heavy load, the 1s default timeout for the Store liveness probe can be often triggered, which leads to Store restarts. The default value is 1s, to keep the current behavior, and in affected environments one can increase it. Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
| scheme: 'HTTP', | ||
| port: ts.config.ports.http, | ||
| path: '/-/healthy', | ||
| }, |
There was a problem hiding this comment.
Umm, maybe we can just make the whole livenessProbe configurable and use ts.config. livenessProbe? In case we want to expose other fields in the future.
There was a problem hiding this comment.
I was torn between making only exactly what I need configurable vs making the whole thing configurable. As a poorly configured liveness probe can cause substantial problems I refrained from making it completely configurable.
I could make the failureThreshold and periodSeconds configurable in addition to the timeoutSeconds. What do you think, @yeya24?
There was a problem hiding this comment.
I feel it is fine to expose the field completely. Since we leave the defaults here so users can take their own risk to change to what they need.
But exposing failureThreshold, periodSeconds and timeoutSeconds is also a way to go. Let's expose these fields for now.
There was a problem hiding this comment.
Done it in 449e423. 👍
Adds configuration for failure threshold and period seconds. Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata [email protected]
Changes
Under heavy load, the 1s default timeout for the Store liveness probe can be often triggered, which leads to Store restarts. This PR makes such timeout configurable.
The default value is 1s, to keep the current behavior, and in affected environments one can increase it.
Verification