Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-bazel-build

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of [email protected]. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-bazel-build to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: Migrate pull-kubernetes-integration to k8s-infra-prow-build #18814

NOTE: migrating this job is not as straightforward as some of the other #18550 issues, because:

• other flags need to be removed to migrate it off of RBE

- --config=remote
- --remote_instance_name=projects/k8s-prow-builds/instances/default_instance

• its behavior off of RBE may be substantially different or consume more resources than with RBE (ref: https://docs.bazel.build/versions/master/remote-execution.html)

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-bazel-build
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-bazel-build&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-bazel-build
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-bazel-build
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-bazel-build for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-bazel-build for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs

[spiffxp]

Suggest a similar strategy to #19070 (comment) to migrate: first try a canary job

See if a CI variant has already been migrated or canaried to see if there has been any impact

[cpanato]

/assign

[rayandas]

/assign

[ameukam]

/assign

[cpanato]

/unassign

[rayandas]

/unassign

[spiffxp]

#19855 merged 2020-11-23 10:29 PT

Holding open to verify jobs continue to pass / haven't changed behavior negatively

[spiffxp]

https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-bazel-build&grid=old&graph-metrics=test-duration-minutes

Bump in duration from ~10min to ~43min is not too surprising, we saw a similar sized jump when migrating pull-kubernetes-bazel-test (ref: #19070 (comment))

Not enough traffic yet to understand if flakiness has increased vs. PR's with broken builds

[spiffxp]

There's been plenty of traffic by this point. Flakiness (or at least the failure rate) has not increased.

Screenshot from local grafana instance pointed at k8s-gubernator:builds: the jump in daily 99p duration is when we switched, and failure rate has trended down since then

Also checked the metrics-explorer graphs for resource usage, the job resources are not over/under-provisioned

Thanks @ameukam!

/close

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

There's been plenty of traffic by this point. Flakiness (or at least the failure rate) has not increased.

Screenshot from local grafana instance pointed at k8s-gubernator:builds: the jump in daily 99p duration is when we switched, and failure rate has trended down since then

Also checked the metrics-explorer graphs for resource usage, the job resources are not over/under-provisioned

Thanks @ameukam!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.