Introduce `krel obs specs` command to generate specs and archives for Open Build Service #2946

xmudrii · 2023-03-06T10:32:28Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR is the first step for implementing KEP-1731 (kubernetes/enhancements#1731).

Introduction

This PR introduces a new subcommand in krel: krel obs specs. This command has two key responsibilities:

Generate RPM spec flies to be uploaded on Open Build Service
Create an archive with binaries that will be used by Open Build Service as a source for binaries in packages

We are intentionally introducing a new command instead of refactoring kubepkg because kubepkg is used in CI and refactoring it would break the CI.

Differences compared to `kubepkg`

krel obs specs doesn't support Debian (.deb) packages
- Technically speaking, krel obs specs can work with Debian specs, but we are not going to maintain those specs any longer in favor of using debbuild (as described in the KEP)
krel obs specs cannot build packages -- we're going to use OBS for this

Workflow

The workflow is as follows:

krel obs specs is going to collect information about packages that we need specs and archives for
- That includes finding the appropriate package versions, Kubernetes versions, and more
krel obs specs is going to generate specs based on given templates
krel obs specs is going to download binaries for selected packages and architectures, then archive those with the maximum compression

Example usage

# Run from the `k/release` root
krel obs specs --kubernetes-version v1.26.1

The command is going to print the output directory after finishing and you can examine that directory for correctness.

TODOs

We need to take care of the following tasks and I propose that we do that in follow up PRs:

Add additional tests, especially around generating specs and downloading archives
Documentation

Which issue(s) this PR fixes:

xref kubernetes/enhancements#1731

Does this PR introduce a user-facing change?

Introduce `krel obs specs` command to generate specs and archives for Open Build Service

/assign @saschagrunert @cpanato @puerco
cc @kubernetes/release-engineering

xmudrii · 2023-03-06T10:48:15Z

/hold
for discussion

saschagrunert

So is my assumption correct that kubepkg will be used to generate the files which can be directly consumed by OBS?

If so, then I think we should build something in parallel to the existing implementation by copying the re-usable code parts and deprecating kubepkg. I think the pkg/kubepkg approach is right here, and we should avoid breaking existing CI solutions.

There are a bunch of locations where the tool, the spec files or documentation around it is already used in CI: https://cs.k8s.io/?q=kubepkg&i=nope&files=&excludeFiles=&repos=

pkg/kubepkg/fetcher.go

saschagrunert · 2023-03-07T09:26:42Z

pkg/kubepkg/fetcher.go

+	return nil
+}
+
+func extractTarGz(gzipStream io.Reader, path string) error {


How about using https://github.com/kubernetes-sigs/release-utils/blob/3532635ef5307586ddfc413f4e39af0fd37fa7c7/tar/tar.go#L113 ?

The problem with this function is that it expects a file, but this function works directly with the response body without writing it to the disk. Is this a blocking comment or can I resolve it in a follow up?

We can surely follow-up on that.

xmudrii · 2023-03-07T09:44:08Z

So is my assumption correct that kubepkg will be used to generate the files which can be directly consumed by OBS?

Yes, kubepkg will generate all files and we just need to osc commit them (and wait for builds to be done).

If so, then I think we should build something in parallel to the existing implementation by copying the re-usable code parts and deprecating kubepkg.

I was thinking about that but I wasn't sure. Let's do it that way then, I'll update the PR later today.

I think the pkg/kubepkg approach is right here, and we should avoid breaking existing CI solutions.

I'm not sure I fully understand this part. Do we want to leave pkg/kubepkg package as it is, but create a new package like pkg/kubepkg2 that's going to be used for the new implementation?

saschagrunert · 2023-03-07T10:01:55Z

So is my assumption correct that kubepkg will be used to generate the files which can be directly consumed by OBS?

Yes, kubepkg will generate all files and we just need to osc commit them (and wait for builds to be done).

We also have to osc checkout and stuff like that, but yeah I think we can have a dedicated library as part of this repository for that. The question is: do we want to use the OBS request feature to submit changes or just commit into the main repositories? For simplicity reasons we can assume that we just commit into them for now.

If so, then I think we should build something in parallel to the existing implementation by copying the re-usable code parts and deprecating kubepkg.

I was thinking about that but I wasn't sure. Let's do it that way then, I'll update the PR later today.

Thank you!

I think the pkg/kubepkg approach is right here, and we should avoid breaking existing CI solutions.

I'm not sure I fully understand this part. Do we want to leave pkg/kubepkg package as it is, but create a new package like pkg/kubepkg2 that's going to be used for the new implementation?

Yes, let's create a new package by copying the code. This way we can follow the KEP by building something in parallel. Maybe it's time to find a better name for it. I'd suggest that we use pkg/obs for everything around managing the OBS repo part, for generating the spec we could still use that namespace like pkg/obs/rpm or something?

We can then use a krel subcommand like krel obs for manual invocation.

xmudrii · 2023-03-07T10:04:52Z

We also have to osc checkout and stuff like that, but yeah I think we can have a dedicated library as part of this repository for that. The question is: do we want to use the OBS request feature to submit changes or just commit into the main repositories? For simplicity reasons we can assume that we just commit into them for now.

I was thinking about just committing them because request feature requires some manual interaction (AFAIK).

Yes, let's create a new package by copying the code. This way we can follow the KEP by building something in parallel. Maybe it's time to find a better name for it. I'd suggest that we use pkg/obs for everything around managing the OBS repo part, for generating the spec we could still use that namespace like pkg/obs/rpm or something?

We can then use a krel subcommand like krel obs for manual invocation.

Sounds like a good plan to me!

xmudrii · 2023-03-07T12:10:44Z

@saschagrunert Ready for another round of reviews.

xmudrii · 2023-03-09T20:30:08Z