-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solutions] Fixes 11 different flakey FTR/e2e tests and scenarios #115688
[Security Solutions] Fixes 11 different flakey FTR/e2e tests and scenarios #115688
Conversation
@@ -344,6 +343,7 @@ export default ({ getService }: FtrProviderContext) => { | |||
], | |||
]); | |||
await waitForRuleSuccessOrStatus(supertest, id); | |||
await waitForSignalsToBePresent(supertest, 4, [id]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NOTE: This waitForSignalsToBePresent
was missing and this is what caused the direct skipping from here:
#115310
@elasticmachine merge upstream |
@@ -499,7 +499,7 @@ export default ({ getService }: FtrProviderContext) => { | |||
], | |||
]); | |||
await waitForRuleSuccessOrStatus(supertest, id); | |||
await waitForSignalsToBePresent(supertest, 1, [id]); | |||
await waitForSignalsToBePresent(supertest, 3, [id]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was just running before all 3 were present before?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is correct, I went ahead and adjusted it. I don't think it failed before but to be consistent with the others I decided to change any of these that were incorrect counts which should reduce chances of flake.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome. Code LGTM!
💚 Build SucceededMetrics [docs]
History
To update your PR or re-run it, just comment with: |
…arios (elastic#115688) ## Summary Fixes flakes across tests that have either been skipped or have been a source of flake in the categories of: * Sorting fixes because Elasticsearch can return hits/arrays back in different orders * Flat array fixes because Elasticsearch can sometimes return `[]` or `[[]]` in-deterministically in some cases 🤷 , so we just flatten the array out completely and test for `[]` within those tests. * `waitForSignalsToBePresent` was missing in a test and sometimes we would get an empty array response which would fail CI. Also I audited other tests for `[[]]` and `waitForSignalsToBePresent` and fixed them where they were present or if the `waitForSignalsToBePresent` count was incorrect. This should give us more stability when the CI is under pressure. Sorting fixes: elastic#115554 elastic#115321 elastic#115319 elastic#114581 Flat array fixes: elastic#89052 elastic#115315 elastic#115308 elastic#115304 elastic#115313 elastic#113418 Missing additional check for "waitForSignalsToBePresent" or incorrect number of signals to wait for fixes: elastic#115310 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
💚 Backport successful
This backport PR will be merged automatically after passing CI. |
…arios (#115688) (#115699) ## Summary Fixes flakes across tests that have either been skipped or have been a source of flake in the categories of: * Sorting fixes because Elasticsearch can return hits/arrays back in different orders * Flat array fixes because Elasticsearch can sometimes return `[]` or `[[]]` in-deterministically in some cases 🤷 , so we just flatten the array out completely and test for `[]` within those tests. * `waitForSignalsToBePresent` was missing in a test and sometimes we would get an empty array response which would fail CI. Also I audited other tests for `[[]]` and `waitForSignalsToBePresent` and fixed them where they were present or if the `waitForSignalsToBePresent` count was incorrect. This should give us more stability when the CI is under pressure. Sorting fixes: #115554 #115321 #115319 #114581 Flat array fixes: #89052 #115315 #115308 #115304 #115313 #113418 Missing additional check for "waitForSignalsToBePresent" or incorrect number of signals to wait for fixes: #115310 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
Summary
Fixes flakes across tests that have either been skipped or have been a source of flake in the categories of:
[]
or[[]]
in-deterministically in some cases 🤷 , so we just flatten the array out completely and test for[]
within those tests.waitForSignalsToBePresent
was missing in a test and sometimes we would get an empty array response which would fail CI.Also I audited other tests for
[[]]
andwaitForSignalsToBePresent
and fixed them where they were present or if thewaitForSignalsToBePresent
count was incorrect. This should give us more stability when the CI is under pressure.Sorting fixes:
#115554
#115321
#115319
#114581
Flat array fixes:
#89052
#115315
#115308
#115304
#115313
#113418
Missing additional check for "waitForSignalsToBePresent" or incorrect number of signals to wait for fixes:
#115310
Checklist