-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments·cy·ts - Alert user assignment - ESS & Serverless Basic rendering "before each" hook for "alert with no assignees in alerts table" "before each" hook for "alert with no assignees in alerts table" #173429
Comments
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
New failure: CI Build - main |
New failure: CI Build - main |
/skip |
Skipped main: 1865d4d |
## Summary These changes fix flaky alert assignments cypress tests: - #173429 - #172520 - #172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
We enabled these tests in #175965 and will be monitoring for new flakes. |
New failure: CI Build - main |
@e40pud would you mind checking if it's the same issue occurring? Should we see if we can move some of these into unit tests? |
@yctercero yeah looks like it is the same issue. It happens in "before each" section, which means we would need to move all of them into unit tests. I want to try to change the way we activate necessary accounts - instead of just login to all of them one by one, I will login and load the main security page. Here is the draft PR, once the flaky test runner passed successfully I will convert it into ready to review PR #176689 |
## Summary These changes fix flaky alert assignments cypress tests: - elastic#173429 - elastic#172520 - elastic#172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
New failure: CI Build - main |
New failure: CI Build - main |
@e40pud is there a way to mock out these profile calls? |
## Summary These changes fix flaky alert assignments cypress tests: - elastic#173429 - elastic#172520 - elastic#172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)
New failure: CI Build - 8.13 |
New failure: CI Build - main |
## Summary Addresses: * #176529 * #172557 * #177573 * #173429 Fix flaky alert assignments tests. I split assignments tests into two groups: tests with one assignee available and tests with multiple assignees. Right now there is a flakiness in tests with multiple assignees. Most probably it is happening because we do multiple login calls in a row to make sure we activate different users to make them available for assignments: ``` // Login into accounts so that they got activated and visible in user profiles list login(ROLES.t1_analyst); login(ROLES.t2_analyst); login(ROLES.t3_analyst); login(ROLES.soc_manager); login(ROLES.detections_admin); login(ROLES.platform_engineer); ``` These tests are tend to be flaky and it is possible that kibana operations team will skip those. To make sure that we run basic cypress verification of alert assignments feature we decided to add tests with only one assignee available (current user) which allows us to avoid multiple consecutive login calls. Also, as part of these changes I removed unnecessary logins and un-skipped #176529 ## NOTE After discussing these failure with the team, we decided to remove tests which are covered by the integration and unit tests. While fixing the flakiness we realised that we do unnecessary work trying to fight the internal errors within elastic search on serverless when we do multiple user logins in a row. Instead we will rely on: * integration tests coverage of API related functionality including RBAC * unit tests coverage of all assignments UI components * cypress tests coverage of basic UI interaction with the alert assignments with only one user available for the assignments cc @yctercero ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 50 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5181) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5182) --------- Co-authored-by: Kibana Machine <[email protected]>
We refactored and enabled these tests in #176930. |
A test failed on a tracked branch
First failure: CI Build - main
The text was updated successfully, but these errors were encountered: