-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[mdb-community] A summary of evaluation results for mongodb-community operator #166
Comments
Thanks @taham0 for the detailed explanation! The high FP rate has been the key challenge of Acto now. In this case, the FP is almost 94% (48/51) which is hard to make a case of a usable tool. @tylergu has been working hard on reducing FP and the FPs are indeed reduced by >2 times. However, it does not seem to be sufficient at this point. I don't know how the results of the other operators look like. My guess is that the numbers are low, as the FP reduction was designed based on the understanding of the other operators (we likely have overfitting issues). |
Thanks for the write up. Let’s setup a synchronous meeting to discuss the results. Some of the false alarms seem to be true alarms, e.g. the 7 alarms caused because operator crashed. @taham0 , did you directly analyze the results after running Acto? This is the results without the static analysis support, I think after applying the static analysis the some false alarms should be gone. A lot of the false alarms seem to due to invalid input, if the invalid input is also indicated on the warning level, then we should also try capture the warning level log The 17 false alarms caused because input getting rejected is a bug in Acto we just discovered recently, Acto should recognize this is an invalid input if any error message appear from kubectl’s stderr I am more curious why the true alarm number has decreased so much, which true alarms can not be reproduced anymore. |
|
That's awesome!! Thank you for all the hard work @taham0 ! |
The FP rate for mongodb-community-operator has reduced to 44.4% (4 / 9 FA) after some improvements and the latest two evaluation results have been included above. All bugs were reproduced and a by-product bug was found. |
Evaluation Results
True Alarms
The initial evaluation had reproduced 4 bugs. A total of 3 true alarms reproduced 2 bugs in the most recent evaluation.
scramCredentialsSecretName
creates a new secretFalse Alarms
The initial evaluation had 108 false alarms. The number of false alarms reduced to 48 in the most recent evaluation.
process
names were invalid.The
spec.automationConfig
contains an array of processes. Eachprocess
has specific fields and corresponding values specified in the input to override the current processes in the operator-createdautomationConfig
by merging. Specifically, the operator searches for the process names specified in the input among the current processes. Since the input process names are invalid, no match is found and the input is not merged and the automationConfig remains unchanged.In order to enable TLS, a
caCertificateSecret
(or acaConfigMap
) and acertificateKeySecret
is required. None of these objects exist and Acto provides an invalid reference to one or more of these objects which is identified by the operator and results in a warning.A previously applied configuration left the operator in an inconsistent state. Since agent
version
does not match the goal state, thereplicaSet
is not ready. Consequently, the operator is unable to proceed towards creating / updating theconnectionStringSecret
.Evaluation Result 28/08/22
The following changes were made:
True Alarms | 5
False Alarms | 13
Evaluation Result 31/08/22
True Alarms | 5
False Alarms | 4
The text was updated successfully, but these errors were encountered: