Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-dependencies

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of [email protected]. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-dependencies to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: Migrate pull-kubernetes-integration to k8s-infra-prow-build #18814

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-dependencies
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-dependencies&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-dependencies
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-dependencies
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-dependencies for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-dependencies for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs
/help

[helenfeng737]

/assign

[helenfeng737]

/remove-help

[spiffxp]

Checking in, how are things looking?

[helenfeng737]

does the job start failing more often?
does the job start going into error state?

No

does the job duration look worse than before? spikier than before?

No. Haven't noticed any irregular pattern.

do more failures show up than before?

No. Even fewer failures in the past 2 weeks.

is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)

We give CPU limit as 2 and in the past 2 weeks the CPU usages are mostly below 1(see diagram below). Probably we can lower the limit.

metrics explorer: Memory limit utilization for pull-kubernetes-dependencies for 6h
is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)

We give memory limit as 1.2G and so far the usage is between 0.6-1G. We can keep this number.

[helenfeng737]

Checking in, how are things looking?

Please see the above.

[spiffxp]

/close
Thanks for the update, I think we can call those done!

I'm fine leaving the CPU where it's at, looking at shorter time ranges it hits ~100% utilization; the larger time windows use larger mean intervals than 1min.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Thanks for the update, I think we can call those done!

I'm fine leaving the CPU where it's at, looking at shorter time ranges it hits ~100% utilization; the larger time windows use larger mean intervals than 1min.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.