Testing: Establishing Load Test Requirements

[RealAnna]

To design a few meaningful test scripts for our load tests we should establish reasonable performance requirements.

DoD

• motivated list of requirements
• first draft script for a load test (maybe in a form of a new testing ticket) - fetching data from prometheus of how many API call per minute to the KubeAPI server

AC

• performance requirements are documented
• first load tests are sketched

[bacherfl]

A rough first implementation of how load tests could be executed can be found in the following draft PR: #599

In this PR, tests are executed with kube-burner, which might be an interesting choice for doing load tests for Operators, and it also provides a straight forward way to integrate with Prometheus.

As far as the requirements goes, we should always ensure the following:

1. Number of API Calls should not significantly increase with number of already reconciled objects
2. Memory Consumption should not significantly increase with number of already reconciled objects
3. CPU Usage should not significantly increase with number of already reconciled objects

So essentially, the resource footprint should roughly stay the same, regardless if we have a fresh installation or a high number of already reconciled CRDs which do not require any action anymore. E.g.., there should be no endless reconciliation loops for already finished Tasks, Evaluations etc., and memory leaks must be avoided.

Also, as a follow up task we should find the breaking point wrt to how many KLT CRDs we can actively reconcile at the same time in a cluster