Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Cypress: create rules disabled by default #147377

Open
Tracked by #161505
banderror opened this issue Dec 12, 2022 · 2 comments
Open
Tracked by #161505

[Security Solution] Cypress: create rules disabled by default #147377

banderror opened this issue Dec 12, 2022 · 2 comments
Labels
performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test

Comments

@banderror
Copy link
Contributor

banderror commented Dec 12, 2022

Epic: #153633
Related to: #147349

Summary

When we create detection rules in Cypress tests, in most cases we create them enabled. This shouldn't be needed for many of the tests, e.g. it's not needed for the majority of rule management tests. Enabled rules consume resources of the BE which negatively affects the performance of Cypress tests themselves.

By default, create rules disabled. Only enable them or create enabled where necessary.

@banderror banderror added performance technical debt Improvement of the software architecture and operational architecture Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Dec 12, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

banderror added a commit that referenced this issue Dec 12, 2022
…147349)

## Summary

This PR makes the following adjustments in the Cypress tests:

- Increases the default rule interval from 1 second to 100 minutes.
- Increases the long-polling interval for the
`api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2
seconds.

This should hopefully reduce the load that Cypress tests generate on CI
on Kibana which makes the tests themselves slower.

In the next episodes:

- #147377

## Background

With the recent issues related to installing the prebuilt rules package,

> @dhurley14 did some initial research and found that there are some
bulk_edit cypress tests that are going into an infinitely loop. Looks
like the infinite loops can be reproduced locally and are happening here
[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)

@xcrzx @maximpn and I reproduced the issue locally and found that:

- There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it
took more than 1 hour to run 3 attempts of a single test: `Only prebuilt
rules selected`.
- 6 rules that this test created were enabled and were executing every 1
second each.
- The `api/detection_engine/rules/prepackaged/_status` endpoint was
being called 600 times per minute until the timeout. It looks like it
took _way_ more than a few minutes to run each attempt.

We were able to reproduce this locally with the exception that the
attempts didn't take much time to fail.

The hypothesis is that on CI where we have less computing power our
Cypress tests overload the Kibana server by creating enabled rules that
execute way too frequently. We shouldn't need rules to be enabled in
many of the tests; we should be able to set a very long rule interval
where rules have to be enabled. Finally, we shouldn't be spamming the
API with highly frequent requests where we implement long-polling and
wait for something to happen on the BE side.
kibanamachine pushed a commit that referenced this issue Dec 20, 2022
…147349)

## Summary

This PR makes the following adjustments in the Cypress tests:

- Increases the default rule interval from 1 second to 100 minutes.
- Increases the long-polling interval for the
`api/detection_engine/rules/prepackaged/_status` endpoint from 0.1 to 2
seconds.

This should hopefully reduce the load that Cypress tests generate on CI
on Kibana which makes the tests themselves slower.

In the next episodes:

- #147377

## Background

With the recent issues related to installing the prebuilt rules package,

> @dhurley14 did some initial research and found that there are some
bulk_edit cypress tests that are going into an infinitely loop. Looks
like the infinite loops can be reproduced locally and are happening here
[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)

@xcrzx @maximpn and I reproduced the issue locally and found that:

- There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but it
took more than 1 hour to run 3 attempts of a single test: `Only prebuilt
rules selected`.
- 6 rules that this test created were enabled and were executing every 1
second each.
- The `api/detection_engine/rules/prepackaged/_status` endpoint was
being called 600 times per minute until the timeout. It looks like it
took _way_ more than a few minutes to run each attempt.

We were able to reproduce this locally with the exception that the
attempts didn't take much time to fail.

The hypothesis is that on CI where we have less computing power our
Cypress tests overload the Kibana server by creating enabled rules that
execute way too frequently. We shouldn't need rules to be enabled in
many of the tests; we should be able to set a very long rule interval
where rules have to be enabled. Finally, we shouldn't be spamming the
API with highly frequent requests where we implement long-polling and
wait for something to happen on the BE side.

(cherry picked from commit 7a8ed4c)
kibanamachine referenced this issue Dec 21, 2022
…tests (#147349) (#147875)

# Backport

This will backport the following commits from `main` to `8.6`:
- [[Security Solution] Reduce load on Kibana generated by Cypress tests
(#147349)](#147349)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Georgii
Gorbachev","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-12-12T17:56:32Z","message":"[Security
Solution] Reduce load on Kibana generated by Cypress tests
(#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments
in the Cypress tests:\r\n\r\n- Increases the default rule interval from
1 second to 100 minutes.\r\n- Increases the long-polling interval for
the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from
0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that
Cypress tests generate on CI\r\non Kibana which makes the tests
themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n-
https://github.com/elastic/kibana/issues/147377\r\n\r\n##
Background\r\n\r\nWith the recent issues related to installing the
prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research
and found that there are some\r\nbulk_edit cypress tests that are going
into an infinitely loop. Looks\r\nlike the infinite loops can be
reproduced locally and are happening
here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx
@maximpn and I reproduced the issue locally and found that:\r\n\r\n-
There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but
it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only
prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were
enabled and were executing every 1\r\nsecond each.\r\n- The
`api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing
called 600 times per minute until the timeout. It looks like it\r\ntook
_way_ more than a few minutes to run each attempt.\r\n\r\nWe were able
to reproduce this locally with the exception that the\r\nattempts didn't
take much time to fail.\r\n\r\nThe hypothesis is that on CI where we
have less computing power our\r\nCypress tests overload the Kibana
server by creating enabled rules that\r\nexecute way too frequently. We
shouldn't need rules to be enabled in\r\nmany of the tests; we should be
able to set a very long rule interval\r\nwhere rules have to be enabled.
Finally, we shouldn't be spamming the\r\nAPI with highly frequent
requests where we implement long-polling and\r\nwait for something to
happen on the BE
side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport","performance","technical
debt","release_note:skip","Team:Detections and Resp","Team:
SecuritySolution","auto-backport","Team:Detection
Rules","v8.6.0","v8.7.0"],"number":147349,"url":"https://github.com/elastic/kibana/pull/147349","mergeCommit":{"message":"[Security
Solution] Reduce load on Kibana generated by Cypress tests
(#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments
in the Cypress tests:\r\n\r\n- Increases the default rule interval from
1 second to 100 minutes.\r\n- Increases the long-polling interval for
the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from
0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that
Cypress tests generate on CI\r\non Kibana which makes the tests
themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n-
https://github.com/elastic/kibana/issues/147377\r\n\r\n##
Background\r\n\r\nWith the recent issues related to installing the
prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research
and found that there are some\r\nbulk_edit cypress tests that are going
into an infinitely loop. Looks\r\nlike the infinite loops can be
reproduced locally and are happening
here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx
@maximpn and I reproduced the issue locally and found that:\r\n\r\n-
There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but
it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only
prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were
enabled and were executing every 1\r\nsecond each.\r\n- The
`api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing
called 600 times per minute until the timeout. It looks like it\r\ntook
_way_ more than a few minutes to run each attempt.\r\n\r\nWe were able
to reproduce this locally with the exception that the\r\nattempts didn't
take much time to fail.\r\n\r\nThe hypothesis is that on CI where we
have less computing power our\r\nCypress tests overload the Kibana
server by creating enabled rules that\r\nexecute way too frequently. We
shouldn't need rules to be enabled in\r\nmany of the tests; we should be
able to set a very long rule interval\r\nwhere rules have to be enabled.
Finally, we shouldn't be spamming the\r\nAPI with highly frequent
requests where we implement long-polling and\r\nwait for something to
happen on the BE
side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"8.6","label":"v8.6.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/147349","number":147349,"mergeCommit":{"message":"[Security
Solution] Reduce load on Kibana generated by Cypress tests
(#147349)\n\n## Summary\r\n\r\nThis PR makes the following adjustments
in the Cypress tests:\r\n\r\n- Increases the default rule interval from
1 second to 100 minutes.\r\n- Increases the long-polling interval for
the\r\n`api/detection_engine/rules/prepackaged/_status` endpoint from
0.1 to 2\r\nseconds.\r\n\r\nThis should hopefully reduce the load that
Cypress tests generate on CI\r\non Kibana which makes the tests
themselves slower.\r\n\r\nIn the next episodes:\r\n\r\n-
https://github.com/elastic/kibana/issues/147377\r\n\r\n##
Background\r\n\r\nWith the recent issues related to installing the
prebuilt rules package,\r\n\r\n> @dhurley14 did some initial research
and found that there are some\r\nbulk_edit cypress tests that are going
into an infinitely loop. Looks\r\nlike the infinite loops can be
reproduced locally and are happening
here\r\n[https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-[…]rity_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts](https://github.com/elastic/kibana/blob/3e56eba64d51aa98ef5d9ed8d7d6cdcfc1f3cd5d/x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts#L166)\r\n\r\n@xcrzx
@maximpn and I reproduced the issue locally and found that:\r\n\r\n-
There's no infinite loop in the `bulk_edit_rules.cy.ts` tests, but
it\r\ntook more than 1 hour to run 3 attempts of a single test: `Only
prebuilt\r\nrules selected`.\r\n- 6 rules that this test created were
enabled and were executing every 1\r\nsecond each.\r\n- The
`api/detection_engine/rules/prepackaged/_status` endpoint was\r\nbeing
called 600 times per minute until the timeout. It looks like it\r\ntook
_way_ more than a few minutes to run each attempt.\r\n\r\nWe were able
to reproduce this locally with the exception that the\r\nattempts didn't
take much time to fail.\r\n\r\nThe hypothesis is that on CI where we
have less computing power our\r\nCypress tests overload the Kibana
server by creating enabled rules that\r\nexecute way too frequently. We
shouldn't need rules to be enabled in\r\nmany of the tests; we should be
able to set a very long rule interval\r\nwhere rules have to be enabled.
Finally, we shouldn't be spamming the\r\nAPI with highly frequent
requests where we implement long-polling and\r\nwait for something to
happen on the BE
side.","sha":"7a8ed4cf866db32fb2f957a6000ddc2474f7cd77"}}]}] BACKPORT-->

Co-authored-by: Georgii Gorbachev <[email protected]>
Co-authored-by: Devin W. Hurley <[email protected]>
Co-authored-by: Pedro Jaramillo <[email protected]>
Co-authored-by: Dmitrii <[email protected]>
@banderror banderror self-assigned this Mar 27, 2023
@banderror banderror removed their assignment May 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test
Projects
None yet
Development

No branches or pull requests

2 participants