CI: no logging by default

[pohly]

The recently added increased logging made the resulting
log files rather large, which is a problem when downloading
them and exhausted available space in the Jenkins master.

Now that extra logging is turned off by default respectively
done less frequently (top and df run once a minute). To enable
logging, edit the pipeline in a PR or in the Jenkins UI.

[pohly]

This worked. Here's a PR test run with the new default settings: https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/view/pmem-csi/job/pmem-csi/view/change-requests/job/PR-445/3 => 7MB log file.

Here's one with all output enabled and 30s delay, done by using the "replay" action for the previous job and editing the script of the pipeline: https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/view/pmem-csi/job/pmem-csi/view/change-requests/job/PR-445/4/ => 15.6MB log file.

I can't reproduce the figure of several GB for a log file that @okartau mentioned. Perhaps that was for a run with severe errors and retries where simply the long runtime with continuous output from the Kubernetes pods caused the log to balloon. That worst-case scenario now cannot happen anymore.

@avalluri, @okartau: Note that triggering a replay with edited script is also a fast way to retry just the failing configuration - simply remove all other steps.

[okartau]

I can't reproduce the figure of several GB for a log file that @okartau mentioned

If you refer to my recent statement in chat, I did not mean "several GB" in context of one build, but overall in CI system. I haven't made measurements myself, I can see that one e2e run locally produces 13..15 MB text file. We have multiple such targets built in every PR. And in case of failure, we have retries. So it does not take so many PR jobs to get to multi GB level.

[pohly]

If you refer to my recent statement in chat, I did not mean "several GB" in context of one build, but overall in CI system. I haven't made measurements myself, I can see that one e2e run locally produces 13..15 MB text file.

I it actually was Chuy who mentioned PR-383 in his email. The largest log for a single job in that PR consumed 6.7G.