Framework for building RPMs in the CI

[webbnh]

This PR contains the various bits of mechanism and tweaks required to build RPMs in containers. (The next step will be to enhance the CI to use the facilities added here -- this doesn't actually hook them up.)

There are several pieces to these changes, but most are small enough and localized enough that it wasn't worth splitting them into separate commits or PRs.

Here are the highlights of the changes:

• agent/Makefile: For no reason which is obvious to me, while the RHEL-9 builds worked fine, the RHEL-8 and RHEL-7 Agent builds failed with a complaint during the SRPM build where we "install" the Python components, because the target location was not a site path nor was it in PYTHONPATH. (Apparently, this doesn't happen when building on or for COPR, but perhaps that's because no one is doing that on RHEL-7/8?) Given that in this context we don't intend to execute the stuff we're installing, this wouldn't be a problem, but the tools object nonetheless. I tried several things to address the problem and ended up simply adding the installation target to the PYTHONPATH (which is empty, otherwise) for the duration of the installation, and that seemed to mollify the tools. Unfortunately, I kind of had to guess where the installation would land, but it's an educated guess, and it seems to work. (Also, I removed the unneeded parentheses -- Make executes each line of a recipe in its own subshell anyway -- and I replaced the ; there with a && which I think better represents our requirements.)
• agent/rpm/pbench-agent.spec.j2: Again, for no reason which is obvious to me, while the RHEL-9 builds worked fine, the RHEL-8 and RHEL-7 Agent builds failed during the RPM build where it byte-compiles all the Python code, because it decided to run Python 2.6 instead of the Python3, because it had to guess. (Again, apparently, this doesn't happen when building on COPR; I don't know why.) The fix is to add a definition to the spec file for __python (we already have one for __python3...) to specify that, when we use Python, we want Python 3.
• server/rpm/Makefile and agent/rpm/Makefile:
  • Add a ci target, for building binary RPM(s) for our favorite distro versions.
    • For the Server, this is just RHEL-9.
    • For the Agent, the list is:
      • RHEL: 7, 8, 9
      • CentOS: 8, 9
      • Fedora: 34, 35, 36
  • Some minor comment copyediting
  • For the Agent, I added some Make magic. Instead of including the common makefile and using the targets directly, it now uses a recursive make invocation. If you're building a single target, there is virtually no difference (I hope!) from the old behavior. However, because the CI wants to build eight logically-independent targets, with this approach it can specify the --jobs flag on the outer make command and all of the RPMs will be built in parallel automagically.
• utils/rpm.mk: This is the real meat of this PR.
  • The principal change is the addition of the %-rpm target, which invokes the make command recursively inside the appropriate specific container to build the generic rpm target. This means that any distribution/version whose tag matches an image label in the container repository can be built, and adding new distro/version can be done without having to modify the makefile simply by pushing new container images.
  • There are a few "details" to note, however. rpmbuild is fairly opinionated about where it gets build inputs from and where it sends its outputs to. This can be controlled by setting the _topdir macro, but it turned out to be awkward to get too fancy with that. So, instead, I set up the container invocation to mount the output directory where rpmbuild expects it. Nevertheless, because I wanted to be able to run multiple RPM builds (sequentially at least, and concurrently at best) without them interfering with each other or overwriting each other, it was necessary to make the input/output directory tree distro/version-specific, so I ended up explicitly specifying _topdir anyway. The net result is that rpmbuild uses ${HOME}/rpmbuild${DIST_SUFFIX}, where the suffix is empty for normal builds and it's set to something like -rhel-9 for distro/version builds; thus, the makefile still behaves as it used to if you're doing a "local" build, and otherwise all of the products are maintained separately for each distro/version.
  • Accordingly, I tweaked the references (and definitions) of the temporary directory and the build directories. For the most part, they are referenced symbolically, and they refer to a distro/version-specific directory as appropriate. I say "for the most part", because the "clean-up" make targets are a little bit "raw-er" in order to be thorough and to make the wildcarding work.
  • Unfortunately, because everything is built separately from the same source, we end up with sequential version numbers on the RPMs, e.g. here are a set of RPMs all built from the same Git checkout (so they all have the same hash), but they have different numbers preceding the g (if we think this is a problem, and I craft a fix for it in another PR):
    • ~/rpmbuild/RPMS/noarch/pbench-agent-0.72.0-77ge34d2b480.noarch.rpm
    • ~/rpmbuild-centos-8/RPMS/noarch/pbench-agent-0.72.0-70ge34d2b480.noarch.rpm
    • ~/rpmbuild-centos-9/RPMS/noarch/pbench-agent-0.72.0-69ge34d2b480.noarch.rpm
    • ~/rpmbuild-fedora-34/RPMS/noarch/pbench-agent-0.72.0-74ge34d2b480.noarch.rpm
    • ~/rpmbuild-fedora-35/RPMS/noarch/pbench-agent-0.72.0-69ge34d2b480.noarch.rpm
    • ~/rpmbuild-fedora-36/RPMS/noarch/pbench-agent-0.72.0-75ge34d2b480.noarch.rpm
    • ~/rpmbuild-rhel-7/RPMS/noarch/pbench-agent-0.72.0-72ge34d2b4.noarch.rpm
    • ~/rpmbuild-rhel-8/RPMS/noarch/pbench-agent-0.72.0-71ge34d2b480.noarch.rpm
    • ~/rpmbuild-rhel-9/RPMS/noarch/pbench-agent-0.72.0-73ge34d2b480.noarch.rpm
  • I had to "dumb down" the definition of VERSION because it unaccountably (and silently) failed on the older versions of RHEL (perhaps because they have an older version of Git?). Likewise, the git submodule command got pissy on the older versions, but that was straightforward to work around, as well.

PBENCH-720

[ndokos]

Needs a rebase now that #2960 is merged.

[webbnh]

Rebased.

[portante]

Let's file a PR to avoid the auto-increment on the sequence number across RPMs.

[webbnh]

Let's file a PR to avoid the auto-increment on the sequence number across RPMs.

PBENCH-872