List of next steps

[tylergu]

Running several test cases at the same time

Currently Acto generates a bunch of test cases at the beginning, and execute the test cases one by one.
Each test case is to change a field in the CR.
One possible optimization is to execute multiple test cases at the same time, which means to mutate multiple fields at once.
If we execute 8 test cases at the same time, and they all pass, it means we used only 1 test run to finish 8 test cases.

However, doing so could have problems if we have a lot of rejected test cases.
If one of the test cases causes an rejection by the operator, then we need to do a binary search to find out which test case caused this rejection so that we can still execute other test cases. If there are a lot of rejection happening, then we will end up doing a lot of binary search and have worse performance.

Domain specific support

While the kubebuilder's controller-gen already provides a lot of kubernetes domain-specific knowledge when generating the CRD, we can develop more domain-specific knowledge when generating inputs so that we can exercise deeper logic of operators.

For example, when CR field requires resource reference to a secret object, we will create a secret object and then reference the secret object there. If the field asks for an image repo, then we need to provide a valid image url.

Although we want to make the input more valid, it is still valuable to test the operators the invalid inputs to exercise its error handling logic.

Back and forth testing

The idea of back and forth testing is that, assuming the declarative nature of operators, if the same input is submitted to operator, then the application deployed should be exactly the same. During testing, we can revert back to an input we submitted before, and compare the two application produced by the operator. If they are not the same, then we find a bug.

Automatically infer benign error messages

The problem is that we realized whenever we port new operators, we always need to write some regular expressions to exclude the benign error messages. The benign error messages are very application-specific so it's hard to use some generic regular expression to capture them all.

There are several possible solutions:

1. One solution is to recognize the error messages that appear in all reconciliation loops. If the same error message appears when we change very different fields, then it means the error message is probably unrelated to the input change.
2. Another idea is to recognize the reconciliation loops in the operator log. If the error messages disappear after several reconciliation loop, then it means this error is transient.

Better invalid input log heuristic

We want to read the operator log to see if the input is valid or invalid. If the operator log shows the input is invalid, then we need to revert to previous input.
We currently do so by doing string matching of the keyword invalid, however, this heuristic is hard to reason. A better heuristic is to check if the error log contains the exact input value.

[tianyin]

please see #132