[Epic] CI Improvements

[fao89]

Author: [email protected] ([email protected])

Redmine Issue: 5762, https://pulp.plan.io/issues/5762

---

Tracking Epic for improvements to the CI, including for non-plugin-template CI like ansible-pulp & pulp-operator.

Problems

There are two problems I can see wit the testing and CI/CD situation for pulpcore and its plugins.

False Negatives

I estimate our tests have a 30% - 50% false negative rate. This is costly in several ways. It slows down throughput of work by having to rerun jobs. Also the jobs don't automatically rerun (and shouldn't) so it waits for a human to be involved. It also significantly reduces the value of our nightly jobs because we ignore them since they are unreliable already. Also it interrupts the release process as it did during the pulp_file 0.1.0 GA release.

Runtime Length

The pulpcore jobs take 22 minutes. The plugins also take many minutes. We need to find a way to shorten the runtime of jobs while still running everything.