Enhancements to the CI pipeline (Part 1, Take 2)

[webbnh]

PBENCH-694

This PR is the second try at the first installment in a series of enhancements to the Pbench CI pipeline. The goal is to have the CI perform lint and style checks, run unit tests, build RPMs, build containers, and run functional tests for each of the Agent, Server, and Dashboard. This PR provides the first of those, running lint and style checks as well as unit tests for the Agent, Server, and Dashboard. (This PR is a split of #2935, deferring the building of RPMs and subsequent stages to a subsequent PR.)

For convenience, this PR consists of a series of commits, but they should be squashed into a single commit when it is merged:

• enhancements to the CI pipeline
• tweaks to the CI dockerfile, adding dependencies of the extended pipeline features
• tweaks to the Dashboard environment for running ESLint
• tweaks to the Dashboard code to make it pass ESlint

Highlights of the changes:

• Add a new top-level script file, build.sh, which executes the steps outlined above. The CI executes this script in the CI container via the jenkins/run command. The intention is that developers should be able to run this script as well, either natively or via a jenkins/run command in their own environment. The script does the following:
  • install the Python and Node.js requirements for running the Agent, Server, and Dashboard linters and the Dashboard unit tests
  • run the various linters
  • run the various unit tests
  • and it will be extended in the future with additional capabilities
• Add some configuration files for ESLint, the Dashboard lint program.
• Make some tweaks to the Dashboard files to enable them to pass the lint checks and unit tests. (I'm hoping that these will come in via another PR from the Dashboard folks...but, if that doesn't happen, then they are here, so that the pipeline won't fail.)
• Tweak the Jenkins pipeline definition to change the invocation from running tox directly to running the new build.sh script
• Remove the responsibility for running the Python linters from Tox, now that it is handled by build.sh.
• Modify the jenkins/run script to add a volume for the ${HOME}/.config directory (so the npm install inside the container can write it), and reorder the volume options in the container invocation from top to bottom (i.e., ${HOME} first, then ${HOME}/.config, and so on).

[dbutenhof]

This seems to have no Jira tag, though the changes themselves look OK. 😦

[webbnh]

This seems to have no Jira tag, though the changes themselves look OK. 😦

Sorry -- I got distracted; I've added it.

[dbutenhof]

This seems to have no Jira tag, though the changes themselves look OK. frowning

Sorry -- I got distracted; I've added it.

While that's nice, the Jira integration doesn't care about the PR description: only the git log messages, the PR title, or the branch name.

[webbnh]

Hmmm...I thought I was following what you did in #2941, but I failed to note that you'd included the tag in the first commit's message. If/when I rebase next, I'll amend it. (In the meantime, I've added the link to Jira manually.)

[dbutenhof]

Hmmm...I thought I was following what you did in #2941, but I failed to note that you'd included the tag in the first commit's message. If/when I rebase next, I'll amend it. (In the meantime, I've added the link to Jira manually.)

Right, the PR description by default is the initial git log message; I usually just squash the newlines to let GitHub wrap it, but otherwise leave it alone.

Adding the link manually to Jira isn't "satisfying" as that doesn't go into the Development pane, and doesn't trigger the automated state change. (Right now we're only automating In Progress when a PR is linked, but there are others we could use, and they're driven off the Development not the general links.)

[ndokos]

When I run ./build.sh in a branch I get tons of npm deprecation warnings which I ignore (although the ads from core-js are a bit harder to ignore), but then I get this:

...
found 13 vulnerabilities (10 moderate, 3 high)
  run `npm audit fix` to fix them, or `npm audit` for details
All done! ✨ 🍰 ✨
224 files would be left unchanged.
/home/nick/src/pbench/github/pbench/worktrees/ws-2946/dashboard
ERROR: invocation failed (exit code 1), logfile: /var/tmp/nick/tox/.package/log/.package-104.log
========================================================================================= log start ==========================================================================================
Error processing line 1 of /var/tmp/nick/tox/.package/lib/python3.9/site-packages/_virtualenv.pth:

  Traceback (most recent call last):
    File "/usr/lib64/python3.9/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named '_virtualenv'

Remainder of file ignored
...

Is that a missing dependency somewhere?

[webbnh]

When I run ./build.sh in a branch I get tons of npm deprecation warnings which I ignore (although the ads from core-js are a bit harder to ignore)

Yeah, I see that too (minus the fancy colors...):

npm WARN @apideck/[email protected] requires a peer of ajv@>=8 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of typescript@>= 2.7 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of react@^15.3.0 || ^16.0.0-alpha but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of react-dom@^15.3.0 || ^16.0.0-alpha but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 1908 packages in 10.717s

202 packages are looking for funding
  run `npm fund` for details

found 24 vulnerabilities (18 moderate, 6 high)
  run `npm audit fix` to fix them, or `npm audit` for details

The "vulnerabilities" statement bothers me, but I've been willfully ignoring it, hoping that @shubham-html-css-js or @MVarshini will tell be that it's "normal" and "nothing to be worried about".

As for the Tox ModuleNotFoundError: No module named '_virtualenv' error, I'm not sure. When I run build.sh, I run it (in a container) like this:

$ jenkins/run ./build.sh

See if that works for you. If so, then there's something missing from your system. (Maybe your Tox is old?)

[ndokos]

Yes, jenkins/run build.sh works. I'll check versions and packages on my system. Thanks!

[shubham-html-css-js]

The "vulnerabilities" statement bothers me, but I've been willfully ignoring it, hoping that @shubham-html-css-js or @MVarshini will tell be that it's "normal" and "nothing to be worried about".

I prefer ignoring these warnings.And I also think this is the best practice to do so.Behind the scenes,The NPM registry runs a security audit on NPM packages. With the release of NPM v6, this command is run automatically when you execute an npm install on your project.
These audit commands takes the current version of package in project and checks the list of known vulnerabilities for that specific package and version.If it finds a vulnerability ,it reports it.

Now should we be worried about these warnings? I prefer not to.In the world of reusable packages, a package can depend on other packages, creating a web of dependencies and the NPM audit command is checking all those dependencies, including those someone else has set up.These are dependencies someone else has added to their packages.If we were to change the dependencies someone else has taken, we cannot expect nothing adverse will happen.Maybe it will work just fine, but it does not sounds like a trivial change.

The best way to fix this would be go the repo of the package, fork it ,modify the dependency and replace it with the fixed version, submit the PR and when the PR gets merged and then use patched version, but I guess that is out of scope for now.

[webbnh]

@dbutenhof, I've rebased the branch and added the Jira tag to the first commit.

should we be worried about these warnings? I prefer not to.

😆 That's my preference, as well, @shubham-html-css-js; but I think we have a requirement to avoid security hazards and other critical bugs, and I'm concerned that ignoring the npm warnings won't help us do that! 🥴

[riya-17]

Looks Good, also it ran smoothly on my system. jenkins/run ./build.sh worked for me.