Publishing Kubernetes packages on community infrastructure

[justaugustus]

Enhancement Description

• One-line enhancement description (can be used as a release note): Publishing Kubernetes packages on community infrastructure
• Kubernetes Enhancement Proposal: keps/sig-release/1731-publishing-packages/README.md
• Discussion Link: https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28
• Primary contact (assignee): @justaugustus @saschagrunert
• Responsible SIGs: SIG Release
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.28
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    Add KEP for publishing packages in provisional state #843
    KEP-1731 : Publishing Kubernetes packages on community infrastructure #3434
    KEP-1731: Extend the KEP with the OBS implementation #3750
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

---

Milestones and Tasks Checklist

Milestone 1.0—Code Deliverable

• Success looks like: debs and rpms are in multiple locations (Google infra and third-party hosting service(s)) @detiber
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.)
  • Describe proposed Flow (what are key actions that the user will take? How will they take them?)
  • Choose the third-party hosting service/Set up a package host of some sort @detiber, @leonardpahlke helping
    • Identified some options to explore
    • Outreach to reps of those options to see if they'll work with us on this WIP
  • Do a spike test to see if the third-party hosting service meets our needs @detiber, @leonardpahlke helping
  • Publish the debs/rpms to the third-party hosting service @detiber

Milestone 1.0—Documentation Deliverable

• Success looks like: Entire deb/rpms move to community infra plan is outlined in a KEP
  • Update this KEP: @RobertKielty (made a PR July 15, 2022)
  • Review the KEP @saschagrunert, @ameukam, @detiber
  • Approve the KEP
  • Get an understanding of the top-level script: what type of infra we're going to run, who pays for this, who can access to this, what type of account we have, tools available @RobertKielty
    - Script reviewed by Robert Kielty and discussed on SIG Release channel with Ben Elder. This KEP will drive outstanding infra/billing questions for the new implementation.
  • Clarify what "top-level script" means (linked in cell above) @RobertKielty
  • Finalize defining the user personas for this work, with group review and sign-off @detiber, @LappleApple

Milestone 1.0—Risk Mitigation

• Success looks like: All risk matters for this milestone are accounted for, plans in place to mitigate
  • Risk: A new GPG key will be issued incompatible with the existing GPG key managed by Google. This is a breaking change for the existing infrastructures using DEB/RPM packages as installation artifacts

Milestone 1.0—Questions resolved

• Resolve if this is needed: Give access to the sandbox env to release eng to work on it @ameukam, @onlydole helping
• Develop sandbox env (GCP project) and give access to Rel Eng to work on it @ameukam, @onlydole helping
• Needed more discussion: Meeting to specifically discuss if we are doing GPG or if we are owning the packages; then who owns new GPG keys and how and when we store them. @ameukam, @onlydole helping
• Generate new keys : Sign kubernetes system packages with GPG release#2627
• Decide on plan: Be able to issue new gcp trusted keys for entire ecosystem/whoever is relying on those packages to deploy/install/upgrade K8s project
• Find/line up people who can build packages for debian, CentOS to work on build tooling (@upodroid to help build Debian packages)

Milestone 2.0—Code deliverable

• Success looks like: implementation (source code) in our repositories, such that we can publish releases by ourselves without having to rely on Google people at all
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.
  • Proposed Flow (what are key actions that the user will take? How will they take them?)
  • Refine the current debs/rpms built by kubepkg
  • Resume discussing how we rethink / rewrite the build tools
  • Rebase the existing kubepkg work into something krel can use directly
  • Krel, the kubernetes release tool, should start including a package build step running the build scripts
  • Phase out publishing to the google controlled host, for having krel publish to our new host for 1-2 releases
  • Users migrate from the Google registry to the community registry, adding the signing key

Milestone 2.0—Documentation Deliverable

• Success looks like: we've published docs on how process works and when it's not working as expected, what to do (for release managers)
  • Expand documentation on the top-level script to drive knowledge transfer
  • Expand documentation on the package build scripts for Debian and RPM to drive knowledge transfer

Milestone 2.0—Questions resolved

• Success looks like: @saschagrunert and @justaugustus have provided context and knowledge about the package build scripts for Debian and rpms
  • Answer: what type of infra we're going to run
  • Answer: who pays for it
  • Answer: what type of account do we have
  • Answer: what tools do/will we have available

Milestone 3.0—Documentation deliverable

• Success looks like: We've updated the google build+publish+release script to download packages uploaded by the upstream release process

Milestone 4.0— Documentation Deliverable

• Success looks like: We've published docs for end users about old infra and new/migrated (more static)
  • Write email to community regarding the required migration to the community repository
  • Send email to community regarding the required migration to the community repository
  • SIG Release informs the community so we don't break people over just one milestone: "these are the keys we'll use to release 1.2X..."

---

Why is this needed?

• It's part of the effort for the community to fully own the infrastructure related to the Kubernetes project.
• Ensure all aspects of project releases can be staffed across companies
• We don’t want to be dependent on a Build Admin any longer (which tends to slow releases)
• We seek more control to eventually choose to build packages for prereleases, extend what packages are shipped, etc.

Why is it needed now?

• The existing infrastructure of this relies on a workstation inside Google offices—essentially one person. It’s not reliable or sustainable.

Who needs it? (User Personas WIP)

• Google Build Team as the existing team building the system packages.
• What does “done” look like/acceptance criteria?
• Automated builds of deb and rpm Kubernetes packages within community infrastructure

Related open issues

• #913

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[xmudrii]

@Rishit-dagli @ramrodo I created placeholder PRs for both docs and feature blog: kubernetes/website#42022 and kubernetes/website#42023

[ruheenaansari34]

Hey again @justaugustus 👋
Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023.

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[xmudrii]

@ruheenaansari34 This is an out of tree KEP, so as of now, it doesn't require any code changes in k/k.

[Atharva-Shinde]

Hey @justaugustus @xmudrii this enhancement is now marked as tracked for the v1.28 Code freeze.

[Rishit-dagli]

Hello @justaugustus @xmudrii wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: kubernetes/website#42022

[sftim]

This is an out of tree KEP, so as of now, it doesn't require any code changes in k/k.

For the alpha, is there any change to document?

[xmudrii]

For the alpha, is there any change to document?

Yes, we plan to mention OBS in docs, but we're still trying to figure out the best way for that.

[sftim]

How about on https://k8s.dev/docs/ - and then revisit for beta?

[xmudrii]

@sftim I'm not sure if that's visible enough. Also, I don't think this KEP will be going through standard alpha/beta/stable criteria, but that's also something to discuss.

[npolshakova]

/remove-label lead-opted-in

[sftim]

This is still labelled alpha; is that appropriate?

[xmudrii]

We're hopefully going to graduate it to beta, I'll see with leads about that

[npolshakova]

Hello @justaugustus @xmudrii, 1.29 Enhancements team here! Is this enhancement targeting 1.29? If it is, can you follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.28 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale