Get rpmbuild to build with Bazel or replace with portable code

[aiuto]

Some people will want to build rpmbuild from source rather than using the system one.
We can facilitate that.

• Create a BUILD file to work with an rpmbuild http_archive
• or.. upstream the BUILD file.

[MarkusTeufelberger]

I'd personally rather not use rpmbuild at all and have bazel assemble the rpm, similar to how deb files are created here.

[aiuto]

I changed the issue description to allow that. It is a hard problem that I don't have time to talk fully about now.

Pro for replacing rpmbuild

• truly portable
• can do any reasonable behavior

Con:

• do we have to replicated rpmbuild behavior feature for feature.

As one example, consider the capability to set default file modes based on the file mode created after you build it from source. This makes sense when the target, execution and host machines are the same, but it is sort of meaningless when they are different. Think about a build cache. Files there might be read-only, but in the source tree they are 644. rpmbuild would normally get the 644, but an artifact produced by Bazel may be 444.

I would welcome a design that allows you to easily and precisely specify what every file mode should be in the RPM, regardless of what it is on disk today. Google does something like this. We have not moved it to Bazel because it requires you to build .deb and .rpm packages in very different ways than you normally think of them - most users would find it too weird. But we could learn from those lessons.

[aiuto]

@sluongng wrote:
building rpm with https://github.com/google/rpmpack

Instead of shelling out and depending on rpmbuild binary from python script https://github.com/bazelbuild/rules_pkg/blob/main/pkg/make_rpm.py,

We can instead relying on the API available in https://github.com/google/rpmpack and construct the runner executable programmatically. More over, not having dependencies on rpmbuild, and instead a golang library, will increase the hermeticity of the rules overall.

This is what currently being used in https://github.com/ericnorris/rules_nfpm which depends on https://github.com/goreleaser/nfpm/ which uses https://github.com/google/rpmpack API.

Adding a dependency on go and an unsupported repository doesn't necessarily make this easier for users. They have more code to install and vet. Do you know how portable rpmpack is to other platforms? Ideally, anything to replace rpmbuild should build on macos and windows.

[sluongng]

Using the nfpm released compiled binary might be the way forward if you don't want tje golang dependencies

[aiuto]

Distributing binaries is an endless treadmill of pain. We would have to build for 3 platforms, then check those in. When someone wants s390 or Apple silicon support, we would have to add new platforms. So that is out.

I just took a second look at https://github.com/google/rpmpack. There are a lot of disclaimers about fitness of purpose and missing capabilities. It just doesn't seem like a capable replacement for rpmbuild.

[sluongng]

1. I think rpmpack, despite its disclaimer, is fairly reliable. We have the entire goreleaser -> nfpm -> rpmpack eco system depending on it, and a lot of projects depending on goreleaser for RPM releases.

2. Looking at https://github.com/rpm-software-management/rpm it seems like rpmbuild is a bunch of C. But perhaps you can vendor this repo in and apply a rpmbuild.BUILD patch on top? Regardless it will be a different set of language dependencies than the current Python setup that you will have to pull in. Personally I would prefer adding and managing the Golang dependencies as its a high level language and attract more contributors/maintainers.

3. A benefit of using nfpm is that it supports both RPM and DEB packaging. So its a good replacement for both RPM and DEB in a unified way.

[nacl]

I am also concerned regarding the compatibility of RPMs not built using the core RPM library. The format is quite old and has all sorts of undocumented compatibility problems that are "solved" in its core libraries. I have been warned about the same from an individual highly familiar with the subject matter.

All that being said, using a non-system hermetic rpmbuild is challenging. My observations:

• Running it on a system without a functioning RPM database requires some setup. I may have some time to document/open source what we did, but that will take some additional time.

• Building rpmbuild from source with bazel may not be feasible without using something like rules_foreign_cc. It has dependencies on multiple third-party libraries that either need to be present on the system or built ourselves, and each of those may have its own dependencies and unique build systems.

  We've avoided this problem internally by using a binary component packaging system to redistribute rpm binaries built outside of bazel, and it all more or less "just works".

When we get to the point of having a generic package target mapping system completely implemented (one part of which is #273), switching between implementations should be much easier and allow users to choose tradeoffs hopefully without changing too much code.

[aiuto]

Closing issues that no one is likely to work on for any forseable future

[kellyma2]

I've been exploring this a bit on my side as I deal with getting the RPMs out that I need to fit the system I'm working within, in particular debuginfo RPMs. To do this in the Fedora/RedHat way requires you to tweak a bunch of RPM variables to match the RPM macros on the underlying system, but this is not necessarily consistent between different systems. For example, my tweaks for Fedora 40 don't appear to work on whatever Ubuntu configured when I installed RPM tools. In addition, debuginfo in particular requires also calls out to a find-debuginfo shell script that is a bit insane and introduces a new dimension of fragility.

TL;Dr is that depending on the system rpmbuild is producing wildly different results on different systems.

In terms of desired approaches here I'm thinking along the lines of:

• generate things using 'enhanced' rules_pkg tailored to Fedora rpmbuild

• build hermetic rpmbuild (maybe using rules_foreign_cc?)

• use a 'driver' model where one of the drivers is the current 'system rpmbuild' one is something like nfpm

@aiuto any strong opinions on this?

[kellyma2]

Also, to clarify, my initial variant is definitely going to just involve getting system rpmbuild to work for debuginfo and testing it on Fedora, CentOS, etc

[aiuto]

any strong opinions?

Mostly about how rpmbuild is an abomination which should be abandoned entirely and replaced with a design that works. But that won't solve any of our current problems.

At this point I have reached the conclusion that rpmbuild is so unpredictable across versions that no changes to pkg_rpm can be truly "correct", even when we have test cases that pass. We can mitigate the danger of touching it in a few ways.

Idea 1: Any PR that adds a new command line arg or environment variable to an rpmbuild call must gate that behind the rpmbuild version. That already shows up in the toolchain info, so we could build from that.

Idea 2: If we are adding new command line options or env variables to support some new attribute (e.g. sub rpms), then it must only be emitted IF that feature is used.

Idea 3: We need to add people who care deeply about RPMs to the project and they are accountable for patching in PRs and ensuring they continue to work on their systems. For example... you want to add debuginfo support. I (and I mean Google internally) will never use that feature. So, I am unlikely to every have an opinion about how easy you make it to use, and we won't be providing any live-in-the-field test cases. More people in the community need to lend effort to add testing support for rpm.

Idea 4: Lots of comments about why any new bits are added to the invocation. We should also start from the assumption that everything we previously did was suspect. I can say with some certainty that the original code was hacked up until something worked on some ancient version of rpmbuild.

[kellyma2]

Have to digest somewhat, but adding some thoughts for posterity :)

any strong opinions?

Mostly about how rpmbuild is an abomination which should be abandoned entirely and replaced with a design that works. But that won't solve any of our current problems.

At this point I have reached the conclusion that rpmbuild is so unpredictable across versions that no changes to pkg_rpm can be truly "correct", even when we have test cases that pass. We can mitigate the danger of touching it in a few ways.

Having explored this a bit more to support the things I (where I = Arista in this case) need my conclusion is that it's not so much an rpmbuild version problem as it is a problem of depending on how rpmbuild macros work and how they vary from system to system. The only solid solution to this is some form of hermetic rpmbuild (and associated pieces). Otherwise we're depending on a teetering pyramid consisting of rpmbuild, system specific rpmbuild macros, debugedit, find-debuginfo.sh, etc. This doesn't seem tenable.

Idea 1: Any PR that adds a new command line arg or environment variable to an rpmbuild call must gate that behind the rpmbuild version. That already shows up in the toolchain info, so we could build from that.

Idea 2: If we are adding new command line options or env variables to support some new attribute (e.g. sub rpms), then it must only be emitted IF that feature is used.

Idea 3: We need to add people who care deeply about RPMs to the project and they are accountable for patching in PRs and ensuring they continue to work on their systems. For example... you want to add debuginfo support. I (and I mean Google internally) will never use that feature. So, I am unlikely to every have an opinion about how easy you make it to use, and we won't be providing any live-in-the-field test cases. More people in the community need to lend effort to add testing support for rpm.

Idea 4: Lots of comments about why any new bits are added to the invocation. We should also start from the assumption that everything we previously did was suspect. I can say with some certainty that the original code was hacked up until something worked on some ancient version of rpmbuild.

Generally agree with what you're describing. I think the big item on my list is going to be related to hermetic use of rpmbuild in the hopes that we can leverage that to reduce some of the variability we're seeing. In the interim, I'm exploring marking the system rpmbuild as platform specific (eg: having the configuration correct for debuginfo for fedora40 or centos7 depending on the user provided config).

[aiuto]

Sounds good. This has to be driven by people using the bleeding edge.