bump kind presubmit jobs resources

[aojea]

Since these jobs are flaking, mainly because pods times out waiting to be running, one guess is that this is due to resources constraint.

With no evidence, maybe try something and observe how it affects is a good approach

xref: #18825 (comment)

[aojea]

/assign @spiffxp @BenTheElder

[spiffxp]

I personally would rather see this done for one job first, whichever is highest traffic, to confirm whether this works

[spiffxp]

https://prow.k8s.io/?type=presubmit&job=*kind*&cluster=k8s-infra-prow-build behavior over the last 12h

[spiffxp]

https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=kubernetes.*kind

There does seem to be a rise in this failure since ~2/19. The k8s-infra-prow-build cluster was updated to use a node-local dns cache add-on, and upgraded to v1.17 at that time (ref: kubernetes/k8s.io#1541 (comment) and kubernetes/k8s.io#1541 (comment))

It's worth noting, this is happening for ci jobs too, which are already at 7 cpu, e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kind-e2e-parallel-1-20/1364237532232945664/prowjob.json

I would try looking into logs on the cluster e.g. resource.type=~"k8s_(cluster|node|pod)" resource.labels.project_id="k8s-infra-prow-build" and trying to filter down to see if you can identify a change in behavior in the last 5 days

[BenTheElder]

I personally would rather see this done for one job first, whichever is highest traffic, to confirm whether this works

+1, also we should be more aggressive for the one job, we should schedule up ~the whole node (7.X) so we can say "yes it works without noisy io neighbors" or "no we still see throttling + pod startup flakes". If it works then we can follow up with less aggressively tuning another job. If it doesn't this is only going to make it harder to schedule and there's some evidence that this may not be sufficient.

We also need to look into kubernetes/k8s.io#1187

[aojea]

/hold
this sounds like a plan

[spiffxp]

~the whole node (7.X)

This has gotten us into trouble almost every time someone does it. Stick to 7 cpu for the test container, the additional overhead of pod utils puts you at 7.4. That's more than enough to prevent other jobs from trying to sneak in.

[BenTheElder]

This has gotten us into trouble almost every time someone does it. Stick to 7 cpu for the test container, the additional overhead of pod utils puts you at 7.4. That's more than enough to prevent other jobs from trying to sneak in.

7.4 / ~8 sounds sufficient, I forgot about the podutils requests.

[BenTheElder]

that particular error string increased ~6 days ago?

kubernetes/kubernetes#99147 (comment) 🙃

[aojea]

that particular error string increased ~6 days ago?

kubernetes/kubernetes#99147 (comment)

hahah, at least we know the reasons, pods are not being able to be running and ready

[spiffxp]

#20966 (comment) - as a reminder, this graph is for CI jobs only, which already have 7 cpu.

kubernetes/k8s.io#1703 raised quota, if you want to modify this to update just pull-kubernetes-e2e-kind I'm willing to merge

[spiffxp]

(images from local grafana dashboard, someday I'll get this public, or setup a datastudio dashboard)

tl;dr I would suggest trying to figure out what changed between 2020-02-04 and 2020-02-07

Is this due to increase in traffic? Probably not. Looking back 6 months, failure rate wasn't this bad pre-holiday lull.

Looking at the last 90 days, for just the kind jobs, it's pretty clear pull-kubernetes-kind-e2e jumped sometime at the beginning of february

Looks like something changed between 2020-02-04 and 2020-02-07

Same story for the CI jobs

[spiffxp]

What I don't understand is why triage doesn't tell the same story:
https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2020-02-11&job=kind

[aojea]

What I don't understand is why triage doesn't tell the same story:
https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2020-02-11&job=kind

it is actually the opposite, the failure rate decreased after

tl;dr I would suggest trying to figure out what changed between 2020-02-04 and 2020-02-07

[aojea]

/hold cancel
just only kind-presubmit

[spiffxp]

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spiffxp]

Triage filters out failures that don't fit into clusters of a certain size by failure text. So it's still the case that the jobs are failing more. Just not due to easily identified common error messages

[k8s-ci-robot]

@aojea: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key kubernetes-kind-presubmits.yaml using file config/jobs/kubernetes/sig-testing/kubernetes-kind-presubmits.yaml

In response to this:

Since these jobs are flaking, mainly because pods times out waiting to be running, one guess is that this is due to resources constraint.

With no evidence, maybe try something and observe how it affects is a good approach

xref: #18825 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.