[Security Solution] Unskip flaky Cypress tests for the Rules table #154694
Closed
Assignees
Labels
8.10 candidate
Feature:Rule Management
Security Solution Detection Rule Management area
impact:high
Addressing this issue will have a high level of impact on the quality/strength of our product.
skipped-test
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
test_ui_functional
test
Comments
spong
added
impact:high
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
banderror
pushed a commit
that referenced
this issue
Apr 11, 2023
## Summary As detailed in #154694, need to address some follow-up flake in addition to #154663. This includes the Rules Table auto-refresh and rule selection suites: [rules_table_auto_refresh.cy.ts](https://github.com/elastic/kibana/blob/ca696ac50c0591acf6723e130d2f9278c2d6ef65/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_table_auto_refresh.cy.ts#L46) Failed builds: * https://buildkite.com/elastic/kibana-pull-request/builds/118556#018762fd-eb9b-4210-a1a7-f28a36e304e7 * https://buildkite.com/elastic/kibana-pull-request/builds/118310#018756a3-4f98-4404-81e5-b55b9644c651 [rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34) Failed builds: * https://buildkite.com/elastic/kibana-pull-request/builds/118310#01875708-6710-47a9-bd3f-892e878bbeb1 * https://buildkite.com/elastic/kibana-pull-request/builds/118700#01876bfe-e766-42b7-a99d-bcdd3c02823a
banderror
changed the title
|
Unskipping tests in rules_selection.cy.ts as part of #155241 Tests in that suite are simple; covering only selection and unselection of rules in the rules table. Flakiness was caused by the installation of 700+ rules from the |
1 task
jpdjere
added a commit
that referenced
this issue
Jul 3, 2023
…t coverage (#155241) ## Summary Extends test coverage for the current Prebuilt Rules installation and update workflows, in the Rules Management area. Follows the test plan: https://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit#heading=h.y4vywfmfu3ef Other changes besides the new tests: - Integration tests related to prebuilt rules were moved to a new `prebuilt_rules` dir from their old `group1` dir. - Existing Cypress tests related to prebuilt rules were renamed to `prebuilt_rules_management.cy.ts` to differentiate those tests to the new tests related to notifications, installation and updates. - Prevented the installation of the +700 prebuilt rules in test suites where it is not necessary. Replaced that with installing a low number of mock prebuilt rules, which enables to test the same functionality. - Unskipping tests in [rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34). See [explanation](#154694 (comment)). ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Jul 3, 2023
…t coverage (elastic#155241) ## Summary Extends test coverage for the current Prebuilt Rules installation and update workflows, in the Rules Management area. Follows the test plan: https://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit#heading=h.y4vywfmfu3ef Other changes besides the new tests: - Integration tests related to prebuilt rules were moved to a new `prebuilt_rules` dir from their old `group1` dir. - Existing Cypress tests related to prebuilt rules were renamed to `prebuilt_rules_management.cy.ts` to differentiate those tests to the new tests related to notifications, installation and updates. - Prevented the installation of the +700 prebuilt rules in test suites where it is not necessary. Replaced that with installing a low number of mock prebuilt rules, which enables to test the same functionality. - Unskipping tests in [rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34). See [explanation](elastic#154694 (comment)). ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit d6d4c64)
kibanamachine
referenced
this issue
Jul 3, 2023
…ow test coverage (#155241) (#161128) # Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] Expand prebuilt rules install/update workflow test coverage (#155241)](#155241) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Juan Pablo Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-03T17:22:40Z","message":"[Security Solution] Expand prebuilt rules install/update workflow test coverage (#155241)\n\n## Summary\r\n\r\nExtends test coverage for the current Prebuilt Rules installation and\r\nupdate workflows, in the Rules Management area.\r\n\r\nFollows the test plan:\r\nhttps://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit#heading=h.y4vywfmfu3ef\r\n\r\nOther changes besides the new tests:\r\n- Integration tests related to prebuilt rules were moved to a new\r\n`prebuilt_rules` dir from their old `group1` dir.\r\n- Existing Cypress tests related to prebuilt rules were renamed to\r\n`prebuilt_rules_management.cy.ts` to differentiate those tests to the\r\nnew tests related to notifications, installation and updates.\r\n- Prevented the installation of the +700 prebuilt rules in test suites\r\nwhere it is not necessary. Replaced that with installing a low number of\r\nmock prebuilt rules, which enables to test the same functionality.\r\n- Unskipping tests in\r\n[rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34).\r\nSee\r\n[explanation](https://github.com/elastic/kibana/issues/154694#issuecomment-1607265120).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d6d4c6495faa77fa8ca093cdc1c92397da1713d1","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","test-coverage","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","ci:no-auto-commit","v8.9.0","v8.10.0"],"number":155241,"url":"https://github.com/elastic/kibana/pull/155241","mergeCommit":{"message":"[Security Solution] Expand prebuilt rules install/update workflow test coverage (#155241)\n\n## Summary\r\n\r\nExtends test coverage for the current Prebuilt Rules installation and\r\nupdate workflows, in the Rules Management area.\r\n\r\nFollows the test plan:\r\nhttps://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit#heading=h.y4vywfmfu3ef\r\n\r\nOther changes besides the new tests:\r\n- Integration tests related to prebuilt rules were moved to a new\r\n`prebuilt_rules` dir from their old `group1` dir.\r\n- Existing Cypress tests related to prebuilt rules were renamed to\r\n`prebuilt_rules_management.cy.ts` to differentiate those tests to the\r\nnew tests related to notifications, installation and updates.\r\n- Prevented the installation of the +700 prebuilt rules in test suites\r\nwhere it is not necessary. Replaced that with installing a low number of\r\nmock prebuilt rules, which enables to test the same functionality.\r\n- Unskipping tests in\r\n[rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34).\r\nSee\r\n[explanation](https://github.com/elastic/kibana/issues/154694#issuecomment-1607265120).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d6d4c6495faa77fa8ca093cdc1c92397da1713d1"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/155241","number":155241,"mergeCommit":{"message":"[Security Solution] Expand prebuilt rules install/update workflow test coverage (#155241)\n\n## Summary\r\n\r\nExtends test coverage for the current Prebuilt Rules installation and\r\nupdate workflows, in the Rules Management area.\r\n\r\nFollows the test plan:\r\nhttps://docs.google.com/document/d/1d_1DYnHlnCaPznWTjeCxhoaRUwxc2O_V0LToAPG0xLE/edit#heading=h.y4vywfmfu3ef\r\n\r\nOther changes besides the new tests:\r\n- Integration tests related to prebuilt rules were moved to a new\r\n`prebuilt_rules` dir from their old `group1` dir.\r\n- Existing Cypress tests related to prebuilt rules were renamed to\r\n`prebuilt_rules_management.cy.ts` to differentiate those tests to the\r\nnew tests related to notifications, installation and updates.\r\n- Prevented the installation of the +700 prebuilt rules in test suites\r\nwhere it is not necessary. Replaced that with installing a low number of\r\nmock prebuilt rules, which enables to test the same functionality.\r\n- Unskipping tests in\r\n[rules_selection.cy.ts](https://github.com/elastic/kibana/blob/3d146298a43e1ba24d83e0ede2758b87e826d0b6/x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts#L34).\r\nSee\r\n[explanation](https://github.com/elastic/kibana/issues/154694#issuecomment-1607265120).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"d6d4c6495faa77fa8ca093cdc1c92397da1713d1"}}]}] BACKPORT--> Co-authored-by: Juan Pablo Djeredjian <[email protected]>
maximpn
added a commit
that referenced
this issue
Aug 11, 2023
…63451) **Addresses:** #154694 ## Summary This PR unskips rules table auto-refresh tests. ## Details Rules table auto-refresh Cypress tests were outdated so it was hard to determine the original flakiness reason. To make it working the following has been done - `mockGlobalClock()` moved above `visit()` to mock the time before visiting the page. It didn't look like working so the test had to wait for 60 seconds (our refresh interval) - removed unnecessary waiting `waitForRulesTableToBeLoaded()` as the test should work without it - removed `setRowsPerPageTo(5)` as the test doesn't look related to the page size - `AutoRefreshButtonComponent ` was refactored in attempt to fix popover closing related flakiness but it doesn't seem to help but the refactoring looks improving readability so leaving as a part of this PR - auto-refresh popover's behavior has been changed. The trigger button gets disabled if there is a rule selected. Clicking the disabled button won't make the auto-refresh popover to appear so this part was changed. ### Encountered issues - `EuiPopover` should close by clicking the trigger button, clicking outside or pressing `Esc` button. The latter two cause flakiness for some reason. I spent quite some time to investigate the reason but without success. It looks like there is a race condition related to [EuiFocusTrap](https://github.com/elastic/eui/blob/main/src/components/focus_trap/focus_trap.tsx) but it doesn't look to be an issue outside Cypress. - `waitForRulesTableToBeLoaded()` basically does nothing after changes in the rules table. Looking at tis contents ```ts export const waitForRulesTableToBeLoaded = () => { // Wait up to 5 minutes for the rules to load as in CI containers this can be very slow cy.get(RULES_TABLE_INITIAL_LOADING_INDICATOR, { timeout: 300000 }).should('not.exist'); }; ``` reveals that `RULES_TABLE_INITIAL_LOADING_INDICATOR` defined as `[data-test-subj="initialLoadingPanelAllRulesTable"]` doesn't exist anymore in our codebase so this function instantly returns. The selector will be fixed in a separate PR. - invoking `mockGlobalClock()` during the test leads to appearing a lot of `Executing a cancelled action` error messages. ### Flaky test runner [50 runs succeeded](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2842)
|
Unskipping rules_table_auto_refresh.cy.ts in #163451. |
maximpn
referenced
this issue
Aug 21, 2023
…ts (#163451) (#164062) # Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] Unskip rules table auto-refresh Cypress tests (#163451)](#163451) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-08-11T12:07:35Z","message":"[Security Solution] Unskip rules table auto-refresh Cypress tests (#163451)\n\n**Addresses:** https://github.com/elastic/kibana/issues/154694\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table auto-refresh tests.\r\n\r\n## Details\r\n\r\nRules table auto-refresh Cypress tests were outdated so it was hard to determine the original flakiness reason. To make it working the following has been done\r\n\r\n- `mockGlobalClock()` moved above `visit()` to mock the time before visiting the page. It didn't look like working so the test had to wait for 60 seconds (our refresh interval)\r\n- removed unnecessary waiting `waitForRulesTableToBeLoaded()` as the test should work without it\r\n- removed `setRowsPerPageTo(5)` as the test doesn't look related to the page size\r\n- `AutoRefreshButtonComponent ` was refactored in attempt to fix popover closing related flakiness but it doesn't seem to help but the refactoring looks improving readability so leaving as a part of this PR\r\n- auto-refresh popover's behavior has been changed. The trigger button gets disabled if there is a rule selected. Clicking the disabled button won't make the auto-refresh popover to appear so this part was changed.\r\n\r\n### Encountered issues\r\n\r\n- `EuiPopover` should close by clicking the trigger button, clicking outside or pressing `Esc` button. The latter two cause flakiness for some reason. I spent quite some time to investigate the reason but without success. It looks like there is a race condition related to [EuiFocusTrap](https://github.com/elastic/eui/blob/main/src/components/focus_trap/focus_trap.tsx) but it doesn't look to be an issue outside Cypress.\r\n- `waitForRulesTableToBeLoaded()` basically does nothing after changes in the rules table. Looking at tis contents\r\n\r\n```ts\r\nexport const waitForRulesTableToBeLoaded = () => {\r\n // Wait up to 5 minutes for the rules to load as in CI containers this can be very slow\r\n cy.get(RULES_TABLE_INITIAL_LOADING_INDICATOR, { timeout: 300000 }).should('not.exist');\r\n};\r\n```\r\n\r\nreveals that `RULES_TABLE_INITIAL_LOADING_INDICATOR` defined as `[data-test-subj=\"initialLoadingPanelAllRulesTable\"]` doesn't exist anymore in our codebase so this function instantly returns. The selector will be fixed in a separate PR.\r\n\r\n- invoking `mockGlobalClock()` during the test leads to appearing a lot of `Executing a cancelled action` error messages.\r\n\r\n\r\n### Flaky test runner\r\n\r\n[50 runs succeeded](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2842)","sha":"589ef228fb9c2525a49d6abc5db64eb98b2c381e","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","skipped-test","release_note:skip","impact:high","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Management","Team:Detection Rule Management","backport:prev-minor","v8.10.0"],"number":163451,"url":"https://github.com/elastic/kibana/pull/163451","mergeCommit":{"message":"[Security Solution] Unskip rules table auto-refresh Cypress tests (#163451)\n\n**Addresses:** https://github.com/elastic/kibana/issues/154694\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table auto-refresh tests.\r\n\r\n## Details\r\n\r\nRules table auto-refresh Cypress tests were outdated so it was hard to determine the original flakiness reason. To make it working the following has been done\r\n\r\n- `mockGlobalClock()` moved above `visit()` to mock the time before visiting the page. It didn't look like working so the test had to wait for 60 seconds (our refresh interval)\r\n- removed unnecessary waiting `waitForRulesTableToBeLoaded()` as the test should work without it\r\n- removed `setRowsPerPageTo(5)` as the test doesn't look related to the page size\r\n- `AutoRefreshButtonComponent ` was refactored in attempt to fix popover closing related flakiness but it doesn't seem to help but the refactoring looks improving readability so leaving as a part of this PR\r\n- auto-refresh popover's behavior has been changed. The trigger button gets disabled if there is a rule selected. Clicking the disabled button won't make the auto-refresh popover to appear so this part was changed.\r\n\r\n### Encountered issues\r\n\r\n- `EuiPopover` should close by clicking the trigger button, clicking outside or pressing `Esc` button. The latter two cause flakiness for some reason. I spent quite some time to investigate the reason but without success. It looks like there is a race condition related to [EuiFocusTrap](https://github.com/elastic/eui/blob/main/src/components/focus_trap/focus_trap.tsx) but it doesn't look to be an issue outside Cypress.\r\n- `waitForRulesTableToBeLoaded()` basically does nothing after changes in the rules table. Looking at tis contents\r\n\r\n```ts\r\nexport const waitForRulesTableToBeLoaded = () => {\r\n // Wait up to 5 minutes for the rules to load as in CI containers this can be very slow\r\n cy.get(RULES_TABLE_INITIAL_LOADING_INDICATOR, { timeout: 300000 }).should('not.exist');\r\n};\r\n```\r\n\r\nreveals that `RULES_TABLE_INITIAL_LOADING_INDICATOR` defined as `[data-test-subj=\"initialLoadingPanelAllRulesTable\"]` doesn't exist anymore in our codebase so this function instantly returns. The selector will be fixed in a separate PR.\r\n\r\n- invoking `mockGlobalClock()` during the test leads to appearing a lot of `Executing a cancelled action` error messages.\r\n\r\n\r\n### Flaky test runner\r\n\r\n[50 runs succeeded](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2842)","sha":"589ef228fb9c2525a49d6abc5db64eb98b2c381e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/163451","number":163451,"mergeCommit":{"message":"[Security Solution] Unskip rules table auto-refresh Cypress tests (#163451)\n\n**Addresses:** https://github.com/elastic/kibana/issues/154694\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table auto-refresh tests.\r\n\r\n## Details\r\n\r\nRules table auto-refresh Cypress tests were outdated so it was hard to determine the original flakiness reason. To make it working the following has been done\r\n\r\n- `mockGlobalClock()` moved above `visit()` to mock the time before visiting the page. It didn't look like working so the test had to wait for 60 seconds (our refresh interval)\r\n- removed unnecessary waiting `waitForRulesTableToBeLoaded()` as the test should work without it\r\n- removed `setRowsPerPageTo(5)` as the test doesn't look related to the page size\r\n- `AutoRefreshButtonComponent ` was refactored in attempt to fix popover closing related flakiness but it doesn't seem to help but the refactoring looks improving readability so leaving as a part of this PR\r\n- auto-refresh popover's behavior has been changed. The trigger button gets disabled if there is a rule selected. Clicking the disabled button won't make the auto-refresh popover to appear so this part was changed.\r\n\r\n### Encountered issues\r\n\r\n- `EuiPopover` should close by clicking the trigger button, clicking outside or pressing `Esc` button. The latter two cause flakiness for some reason. I spent quite some time to investigate the reason but without success. It looks like there is a race condition related to [EuiFocusTrap](https://github.com/elastic/eui/blob/main/src/components/focus_trap/focus_trap.tsx) but it doesn't look to be an issue outside Cypress.\r\n- `waitForRulesTableToBeLoaded()` basically does nothing after changes in the rules table. Looking at tis contents\r\n\r\n```ts\r\nexport const waitForRulesTableToBeLoaded = () => {\r\n // Wait up to 5 minutes for the rules to load as in CI containers this can be very slow\r\n cy.get(RULES_TABLE_INITIAL_LOADING_INDICATOR, { timeout: 300000 }).should('not.exist');\r\n};\r\n```\r\n\r\nreveals that `RULES_TABLE_INITIAL_LOADING_INDICATOR` defined as `[data-test-subj=\"initialLoadingPanelAllRulesTable\"]` doesn't exist anymore in our codebase so this function instantly returns. The selector will be fixed in a separate PR.\r\n\r\n- invoking `mockGlobalClock()` during the test leads to appearing a lot of `Executing a cancelled action` error messages.\r\n\r\n\r\n### Flaky test runner\r\n\r\n[50 runs succeeded](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2842)","sha":"589ef228fb9c2525a49d6abc5db64eb98b2c381e"}}]}] BACKPORT--> Co-authored-by: Kibana Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Caused by: #154695
Summary
Need to address some follow-up flake in addition to #154663. This includes the Rules Table auto-refresh and rule selection suites:
rules_table_auto_refresh.cy.ts
Failed builds:
rules_selection.cy.ts
Failed builds:
The text was updated successfully, but these errors were encountered: