Reducing Kubernetes Build Maintenance

[BenTheElder]

Enhancement Description

• One-line enhancement description (can be used as a release note): Reduce Kubernetes build maintenance by moving to a single build system.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-testing/2420-reducing-kubernetes-build-maintenance/
• Discussion Link: Please remove bazel based build infrastructure kubernetes#88553 (initial community conversation, will circle back to surface the rest)
• Primary contact (assignee): @BenTheElder
• Responsible SIGs: SIG-Testing, SIG-Release
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.21
  • Beta release target (x.y): 1.21
  • Stable release target (x.y): 1.23
• Alpha
  • KEP (k/enhancements) update PR(s): add KEP 2420: Reducing Kubernetes Build Maintenance #2421 Update Reducing Kubernetes Build Maintenance to implementable #2469
  • Code (k/k) update PR(s): Remove Bazel kubernetes#99561 (main PR, full list in Reducing Kubernetes Build Maintenance #2420 (comment), it's ... long)
  • Docs (k/website) update PR(s): N/A? (developer docs will be updated, but not k/website)
• Beta
  • KEP (k/enhancements) update PR(s): update reducing kubernetes build maintenance status #2597
  • Code (k/k) update PR(s): N/A (same as alpha)
  • Docs (k/website) update(s): N/A (not user facing)
• Stable
  • KEP (k/enhancements) update PR(s): mark reducing build maintenance implemented #3209
  • Code (k/k) update PR(s): N/A (same as alpha)
  • Docs (k/website) update(s): N/A (not user facing)

Additional detailed PR listing below #2420 (comment)

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/assign
/sig testing
/sig release

[spiffxp]

Initial draft PR: #2421

[BenTheElder]

followup for implementable #2469

[BenTheElder]

Discussed today in SIG Testing meeting, no concerns there. (this has also been circulated in SIG Testing and Release previously).

[BenTheElder]

UPDATE 02/24/2021

We moved pull-kubernetes-e2e-kind-ipv6 to build without bazel, so far so good. kubernetes/test-infra#21009

We're now running pull-kubernetes-unit (non blocking) which runs ~make test with the race detector enabled as an approximate equivilant to pull-kubernetes-bazel-test (our heaviest bazel dependency in CI). This is working well. kubernetes/test-infra#21008

Nominally speaking we are on track to not depend on this build in CI, opening the path to notify and remove.

---

However there are a handful of test targets in pull-kubernetes-bazel-test that are not golang unit tests and therefore will not be covered by make test.
I produced the following list by inspecting the junit output and ignoring :go_default_test targets (go unit tests under bazel).

• 1. //cluster:common_test

Per BUILD:

# These tests just verify that bash can interpret the file.

This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
EDIT: kubernetes/kubernetes#99486 to remove. See details there.

• 2. //cluster:kube-util_test

Per BUILD:

# These tests just verify that bash can interpret the file.

This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
EDIT: kubernetes/kubernetes#99486 to remove. See details there.

• 3. //hack/boilerplate:boilerplate_test IMPORTANT

Python script that ensures the boilerplate headers are in place.

This one is already covered by pull-kubernetes-verify / make verify so we don't need to do anything.

• 4. //pkg/util:verify-util-pkg

This is a sh_test that basically ensures that ./pkg/util/ only contains code in subdirectories, not ./pkg/util/foo.go

I'm not sure why we couldn't just let approvers handle this, it seems a bit heavyweight, but if we still need it we can move it to pull-kubernetes-verify / make verify.
EDIT: Another approach we could take here is to use prow's blockade plugin to block ^pkg/util/[^/]*\.go by automatically adding do-not-merge/blocked-paths label and posting a configurable explanation comment.
EDIT: Fixed by kubernetes/test-infra#21037 (blockade)
EDIT: kubernetes/kubernetes#99486 to remove.

See: kubernetes/kubernetes#15634

• 5. //test/conformance:conformance_test IMPORTANT

This is a sh_test that ensures the generated conformance.yaml is up to date.

We should probably rework this to run under make verify / pull-kubernetes-verify.
EDIT: fixed in kubernetes/kubernetes#99549 to work under make verify and hack/update-conformance-yaml.sh

[dims]

cc @navidshaikh

[BenTheElder]

Update 02/25/2021

1. pull-kubernetes-unit is looking pretty good https://prow.k8s.io/?job=pull-kubernetes-unit
   pkg/controlplane sometimes times out at the current 240s limit we're setting for make test, we could raise it, but we should fix that test Umbrella issue: slow/timing-out unit tests kubernetes#98486

=== FAIL: pkg/controlplane  (0.00s)
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.
FAIL	k8s.io/kubernetes/pkg/controlplane	240.803s

2. pull-kubernetes-e2e-kind-ipv6 is working fine building with make with ~30m total runtime https://prow.k8s.io/?job=pull-kubernetes-e2e-kind-ipv6. Writing a PR to switch more jobs currently.

3. Filed add blockade to kubernetes for pkg/util test-infra#21037 for 4) on the list above Reducing Kubernetes Build Maintenance #2420 (comment), checking that one off.

4. Filed Cleanup old test targets kubernetes#99486 to cleanup 1) 2) 4) from above.

[aojea]

/cc

[spiffxp]

/priority important-longterm

[BenTheElder]

n.b.: #2420 (comment) is a note from the bi-weekly SIG testing meeting, so I agree with my own comment ;-)

just waiting for 1.19 turndown from SIG release

[BenTheElder]

More cleanup PRs:

• config/jobs: remove some defunct sig-testing-canaries jobs test-infra#23847 (@spiffxp)
• cleanup lingering bazel related scripts/references kubernetes#106099
• removed unused test/e2e_kubeadm/runner/local/run_local.go kubernetes#106100

We are on-track to stable with the turn down of 1.19 related Kubernetes project resources. There's a bit more to do but we're nearly done.

Next up: We run a reasonably large custom bazel cache (greenhouse) instance in each CI cluster primarily aimed at Kubernetes/Kubernetes, we should either turn this down or at least reduce the capacity / operational costs significantly.

[kevindelgado]

Thanks for the update @BenTheElder! Still gonna give you the copy/pasta with all the deets:

Hi @BenTheElder 👋

Checking in once more as we approach 1.23 code freeze at 6:00 pm PST on Tuesday, November 16.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.
• Have a documentation placeholder PR open by Thursday, November 18.

As always, we are here to help should questions come up.

Thanks!!

[spiffxp]

We're past code freeze and test freeze at this point, I'm hoping / assuming nothing remains to be done in kubernetes/kubernetes itself. What tasks do we need to complete to call this done?

[BenTheElder]

There's some pretty minor stuff but we could probably close it out now that we've done the rest.

• turn down greenhouse test-infra#24247
• there's still a few errant references in kubernetes/kubernetes, off the top of my head the publishing bot is still find-and-removing bazel related files still, because we took longer to turn down the 1.19 branch (and also missed turning up 1.23 ...)

[BenTheElder]

none of those references are in production release binary code or release / build scripts though, just in tests, publishing bot config, that sort of thing.

[gracenng]

Hi @BenTheElder ,

1.24 Enhancements Lead here. Could you please update this enhancements's KEP status to implemented, then close this issue?

Thanks :)

[BenTheElder]

filed #3209, we can follow up with any additional minor cleanup.

[BenTheElder]

@gracenng sorry for the delay, this is done!