-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Cypress: create rules disabled by default #147377
Labels
performance
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
technical debt
Improvement of the software architecture and operational architecture
test_ui_functional
test
Comments
banderror
added
performance
technical debt
Improvement of the software architecture and operational architecture
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Team:Detection Rule Management
Security Detection Rule Management Team
labels
Dec 12, 2022
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
banderror
added a commit
that referenced
this issue
Dec 12, 2022
…147349) ## Summary This PR makes the following adjustments in the Cypress tests: - Increases the default rule interval from 1 second to 100 minutes. - Increases the long-polling interval for the `api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2 seconds. This should hopefully reduce the load that Cypress tests generate on CI on Kibana which makes the tests themselves slower. In the next episodes: - #147377 ## Background With the recent issues related to installing the prebuilt rules package, > @dhurley14 did some initial research and found that there are some bulk_edit cypress tests that are going into an infinitely loop. Looks like the infinite loops can be reproduced locally and are happening here [https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166) @xcrzx @maximpn and I reproduced the issue locally and found that: - There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it took more than 1 hour to run 3 attempts of a single test: `Only prebuilt rules selected`. - 6 rules that this test created were enabled and were executing every 1 second each. - The `api/detection_engine/rules/prepackaged/_status` endpoint was being called 600 times per minute until the timeout. It looks like it took _way_ more than a few minutes to run each attempt. We were able to reproduce this locally with the exception that the attempts didn't take much time to fail. The hypothesis is that on CI where we have less computing power our Cypress tests overload the Kibana server by creating enabled rules that execute way too frequently. We shouldn't need rules to be enabled in many of the tests; we should be able to set a very long rule interval where rules have to be enabled. Finally, we shouldn't be spamming the API with highly frequent requests where we implement long-polling and wait for something to happen on the BE side.
kibanamachine
pushed a commit
that referenced
this issue
Dec 20, 2022
…147349) ## Summary This PR makes the following adjustments in the Cypress tests: - Increases the default rule interval from 1 second to 100 minutes. - Increases the long-polling interval for the `api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2 seconds. This should hopefully reduce the load that Cypress tests generate on CI on Kibana which makes the tests themselves slower. In the next episodes: - #147377 ## Background With the recent issues related to installing the prebuilt rules package, > @dhurley14 did some initial research and found that there are some bulk_edit cypress tests that are going into an infinitely loop. Looks like the infinite loops can be reproduced locally and are happening here [https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166) @xcrzx @maximpn and I reproduced the issue locally and found that: - There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it took more than 1 hour to run 3 attempts of a single test: `Only prebuilt rules selected`. - 6 rules that this test created were enabled and were executing every 1 second each. - The `api/detection_engine/rules/prepackaged/_status` endpoint was being called 600 times per minute until the timeout. It looks like it took _way_ more than a few minutes to run each attempt. We were able to reproduce this locally with the exception that the attempts didn't take much time to fail. The hypothesis is that on CI where we have less computing power our Cypress tests overload the Kibana server by creating enabled rules that execute way too frequently. We shouldn't need rules to be enabled in many of the tests; we should be able to set a very long rule interval where rules have to be enabled. Finally, we shouldn't be spamming the API with highly frequent requests where we implement long-polling and wait for something to happen on the BE side. (cherry picked from commit 7a8ed4c)
kibanamachine
referenced
this issue
Dec 21, 2022
…tests (#147349) (#147875) # Backport This will backport the following commits from `main` to `8.6`: - [[Security Solution] Reduce load on Kibana generated by Cypress tests (#147349)](#147349) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-12-12T17:56:32Z","message":"[Security Solution] Reduce load on Kibana generated by Cypress tests (#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments in the Cypress tests:\r\n\r\n- Increases the default rule interval from 1 second to 100 minutes.\r\n- Increases the long-polling interval for the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that Cypress tests generate on CI\r\non Kibana which makes the tests themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n- https://github.com/elastic/kibana/issues/147377\r\n\r\n## Background\r\n\r\nWith the recent issues related to installing the prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research and found that there are some\r\nbulk_edit cypress tests that are going into an infinitely loop. Looks\r\nlike the infinite loops can be reproduced locally and are happening here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx @maximpn and I reproduced the issue locally and found that:\r\n\r\n- There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were enabled and were executing every 1\r\nsecond each.\r\n- The `api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing called 600 times per minute until the timeout. It looks like it\r\ntook _way_ more than a few minutes to run each attempt.\r\n\r\nWe were able to reproduce this locally with the exception that the\r\nattempts didn't take much time to fail.\r\n\r\nThe hypothesis is that on CI where we have less computing power our\r\nCypress tests overload the Kibana server by creating enabled rules that\r\nexecute way too frequently. We shouldn't need rules to be enabled in\r\nmany of the tests; we should be able to set a very long rule interval\r\nwhere rules have to be enabled. Finally, we shouldn't be spamming the\r\nAPI with highly frequent requests where we implement long-polling and\r\nwait for something to happen on the BE side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport","performance","technical debt","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","auto-backport","Team:Detection Rules","v8.6.0","v8.7.0"],"number":147349,"url":"https://github.com/elastic/kibana/pull/147349","mergeCommit":{"message":"[Security Solution] Reduce load on Kibana generated by Cypress tests (#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments in the Cypress tests:\r\n\r\n- Increases the default rule interval from 1 second to 100 minutes.\r\n- Increases the long-polling interval for the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that Cypress tests generate on CI\r\non Kibana which makes the tests themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n- https://github.com/elastic/kibana/issues/147377\r\n\r\n## Background\r\n\r\nWith the recent issues related to installing the prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research and found that there are some\r\nbulk_edit cypress tests that are going into an infinitely loop. Looks\r\nlike the infinite loops can be reproduced locally and are happening here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx @maximpn and I reproduced the issue locally and found that:\r\n\r\n- There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were enabled and were executing every 1\r\nsecond each.\r\n- The `api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing called 600 times per minute until the timeout. It looks like it\r\ntook _way_ more than a few minutes to run each attempt.\r\n\r\nWe were able to reproduce this locally with the exception that the\r\nattempts didn't take much time to fail.\r\n\r\nThe hypothesis is that on CI where we have less computing power our\r\nCypress tests overload the Kibana server by creating enabled rules that\r\nexecute way too frequently. We shouldn't need rules to be enabled in\r\nmany of the tests; we should be able to set a very long rule interval\r\nwhere rules have to be enabled. Finally, we shouldn't be spamming the\r\nAPI with highly frequent requests where we implement long-polling and\r\nwait for something to happen on the BE side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"8.6","label":"v8.6.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/147349","number":147349,"mergeCommit":{"message":"[Security Solution] Reduce load on Kibana generated by Cypress tests (#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments in the Cypress tests:\r\n\r\n- Increases the default rule interval from 1 second to 100 minutes.\r\n- Increases the long-polling interval for the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that Cypress tests generate on CI\r\non Kibana which makes the tests themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n- https://github.com/elastic/kibana/issues/147377\r\n\r\n## Background\r\n\r\nWith the recent issues related to installing the prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research and found that there are some\r\nbulk_edit cypress tests that are going into an infinitely loop. Looks\r\nlike the infinite loops can be reproduced locally and are happening here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx @maximpn and I reproduced the issue locally and found that:\r\n\r\n- There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were enabled and were executing every 1\r\nsecond each.\r\n- The `api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing called 600 times per minute until the timeout. It looks like it\r\ntook _way_ more than a few minutes to run each attempt.\r\n\r\nWe were able to reproduce this locally with the exception that the\r\nattempts didn't take much time to fail.\r\n\r\nThe hypothesis is that on CI where we have less computing power our\r\nCypress tests overload the Kibana server by creating enabled rules that\r\nexecute way too frequently. We shouldn't need rules to be enabled in\r\nmany of the tests; we should be able to set a very long rule interval\r\nwhere rules have to be enabled. Finally, we shouldn't be spamming the\r\nAPI with highly frequent requests where we implement long-polling and\r\nwait for something to happen on the BE side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77"}}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <[email protected]> Co-authored-by: Devin W. Hurley <[email protected]> Co-authored-by: Pedro Jaramillo <[email protected]> Co-authored-by: Dmitrii <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
performance
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
technical debt
Improvement of the software architecture and operational architecture
test_ui_functional
test
Epic: #153633
Related to: #147349
Summary
When we create detection rules in Cypress tests, in most cases we create them enabled. This shouldn't be needed for many of the tests, e.g. it's not needed for the majority of rule management tests. Enabled rules consume resources of the BE which negatively affects the performance of Cypress tests themselves.
By default, create rules disabled. Only enable them or create enabled where necessary.
The text was updated successfully, but these errors were encountered: