Enhancements to the CI pipeline, part 1

[webbnh]

This PR is the first installment in a series of enhancements to the Pbench CI pipeline. The goal is to have the CI perform lint and style checks, run unit tests, build RPMs, build containers, and run functional tests for each of the Agent, Server, and Dashboard. This PR provides the first several of those, up through building RPMs for the Agent and Server (the Dashboard is expected to be deployed using a different mechanism). It also enables the Cobertura report to find and display the source lines!

For convenience, this PR consists of a series of commits, but they should be squashed into a single commit when it is merged (NOTE: this PR is now based on #2938 and #2939, so their respective commits will appear in this PR until they are merged):

• enhancements to the CI pipeline
• tweaks to the CI dockerfile, adding dependencies of the extended pipeline features
• tweaks to the Dashboard environment for running ESLint
• tweaks to the Server RPM build to limit its default build to RHEL-9
• tweaks to the Dashboard code to make it pass ESlint

Highlights of the changes:

• Add a new top-level file, build.sh which executes the steps outlined above. The CI executes this script in the CI container via the jenkins/run command. The intention is that developers should be able to run the script as well, either natively or via a jenkins/run command in their own environment. The script does the following:
  • extract credentials from environment variables and from a local COPR config file
  • install the Python and Node.js requirements for running the Agent, Server, and Dashboard linters and the Dashboard unit tests
  • run the various linters
  • run the various unit tests
  • use COPR (either the internal or the external one) to build the RPMs for the Agent and the Server
  • declare victory (there is latent code for building and pushing the Agent container images and the Pbench Server-in-a-Can container images, but it doesn't work inside the CI container, currently)
• Add support for a "CI" tag for building Agent containers, to keep them separate from the normally published containers.
• Add some configuration files for ESLint, the Dashboard lint program.
• Make some tweaks to the Dashboard files to enable them to pass the lint checks and unit tests. (I'm hoping that these will come in via another PR from the Dashboard folks...but, if that doesn't happen, then they are here, so that the pipeline won't fail.)
• Tweak the Jenkins pipeline definition:
  • add environment variables/secrets for the credentials for accessing COPR and the container registry
  • alphabetize the environment variable list, now that it is long
  • change the invocation from running tox directly to running the new build.sh script
  • add (and remove) a symlink which enables the Cobertura report to find the sources properly
• Remove the responsibility for running the Python linters from Tox, now that it is handled by build.sh, and add to the dependences for the linters in the CI container image definition.
• Add the dependencies for building the RPMs and container images to the CI container image definition.
• Modify the jenkins/run script to add a mapping for the COPR config file from the Jenkins secret file (or the user's file) to the conventional file inside the container. Also, add a volume for the ${HOME}/.config directory (so the npm install inside the container can write it), and reorder the volume options in the container invocation from top to bottom (i.e., ${HOME} first, then ${HOME}/.config, and so on).
• Tweak the Server RPM makefile to set the default build to (only) RHEL-9.
• Change the common RPM makefile to explicitly specify a COPR config file for credentials, and add a copr-ci target.

[dbutenhof]

Cool!

[webbnh]

OK, it seems to be working fairly well, so I think it's ready for review! Thanks!! (And, see you Monday....)

[portante]

Why do we need everything performed in one container?

Why can't the pipeline be broken up into steps, some containerized, some not?

We can also have Jenkins start a particular container environment for us instead of running one monolithic one.

[webbnh]

I believe that we should have everything performed by one command. Now whether that command runs a single script which invokes a series of containers (or whatnot), or whether than command invokes a single container which executes a series of actions is certainly open for discussion. But, what I think we want to avoid is putting lots of stages in the Pipeline.gy file, because then it won't be feasible for developers to run "the CI tests" locally.

The advantage of running it all in a single container (or, perhaps, running each of "The Things" in containers) is that we (collectively and individually) don't have to be concerned about what the respective environments look like. And, if we could run it all in one container, then that would greatly reduce the amount of administration required to run the test pipeline.

But, no, it doesn't have to be that way. It just seemed, at the outset, like a good idea. I would be happy to hear your thoughts on the matter.

[portante]

This change means we don't get these checks in local tox runs. Isn't that bad?

[webbnh]

Why would that be bad?

I mentioned above the desire to have the pipeline run by a single command, but that command doesn't necessarily have to be tox. We have a series of steps -- linters, unit tests, rpm builds, container builds, and perhaps functional tests -- how many of those would you expect to run under tox? Would you expect that Dashboard tests, which have nothing to do with Python be run under tox? (Is it important to have the Python linters run under a particular version of Python, i.e. which Tox provides?)

I'm suggesting that we "switch" the one command from tox to ./build.sh. Would that fit your requirements?

[portante]

Why is this in the public repo?

[webbnh]

Well...I guess that's a good question.

The logic goes like, "The CI container needs those certs, so they are installed as part of the container recipe, which we currently store in the public repo...." But, yes, that's an internal network name, reachable only through the tunnel.

So, how should we proceed?

[portante]

Why does the CI container need those certs?

[portante]

Why not add this to tox?

[webbnh]

Why should it be? See my previous.

[portante]

Let's move the coverage fix to a separate PR.

[webbnh]

I've converted this PR back to draft mode while I attack the following issues:

• enable running containers inside the container used to run build.sh: aside from facilitating the running of functional tests, this will enable us to do containerized RPM builds
• replace the use of COPR with local containerized RPM builds
• remove RH-internal references from these changes (which, hopefully, won't be necessary if we're not using COPR)

@portante and @dbutenhof, please resolve any of your outstanding conversations which are no longer an issue for this PR (e.g., Dave, this PR no longer deals with Cobertura, so the coverage report issues will be addressed in a different PR).

In the meantime, I think I'm going to split off the building of the RPMs to a subsequent PR and limit this PR to just lint, unit tests, and maybe source-RPMs. The additional issues I mentioned at the start here will show up in a subsequent PR.

[webbnh]

This PR has been superseded by PRs #2946, #2982, and follow-ons.