Add failing health check if autoscaler loop consistently returns error #90
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
| } | ||
|
|
||
| func TestNoTimeoutFailingServeHTTP(t *testing.T) { | ||
| w := getTestResponse(time.Now().Add(time.Second*-2), time.Second, time.Second, true) |
There was a problem hiding this comment.
Above 2 tests seem to be exactly identical.
| hc.mutex.Lock() | ||
| defer hc.mutex.Unlock() | ||
| if timestamp.After(hc.lastSuccessfulRun) { | ||
| hc.lastSuccessfulRun = timestamp |
There was a problem hiding this comment.
It probably doesn't matter all that much if we have reasonable timeouts, but maybe we should update lastActivity in here as well? My reasoning is that successfully finishing a loop means that it is running and we're about to do a 10s sleep. So theoretically if a loop took 9:55 we would fail in the middle of the sleep.
|
I wonder how long does it take for an HTTP request to timeout? I'm guessing we don't override default timeouts and I'm wondering how long a single loop will take if we lose connectivity to API server / cloud provider? Maybe it would be worth checking out to make sure that we should get at least 2 or 3 failed loops before failing the probe? |
aleksandra-malinowska
force-pushed
the
health-check-errors
branch
from
cbb180e to
9727724
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
…ed-event UPSTREAM: <carry>: openshift: record max-nodes-total event
NamespaceSelector for ClusterQueue
Extend HealthCheck to verify that a successful run of autoscaler loop occured within given (configurable by setting --max-failing-time flag) time.