Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Triage and skip new flakes in Rule Management tests #173469

Closed
6 tasks done
Tracked by #153633
banderror opened this issue Dec 15, 2023 · 3 comments · Fixed by #173730
Closed
6 tasks done
Tracked by #153633

[Security Solution] Triage and skip new flakes in Rule Management tests #173469

banderror opened this issue Dec 15, 2023 · 3 comments · Fixed by #173730
Assignees
@banderror
Labels
8.12 candidate Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test test-failure-flaky

Comments

@banderror
Copy link
Contributor

banderror commented Dec 15, 2023 •
edited
Loading

Summary

@elastic/security-detection-rule-management, we've got a few new tickets about flakiness in the tests that we own.

The purpose of this ticket is to track these tickets, triage them, and skip the tests that became flaky.

After all these tests are skipped, we can start looking into these tests and fixing them. Until this is done, we should keep those tickets open.

Tickets

Prebuilt Rules
Preview Give feedback
  1. Failing test: Detection Engine Serverless / Prebuilt Rules Management API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/management/fleet_integration·ts - Detection Engine API - Prebuilt Rules Management @ess @serverless @skipInQA install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage #172107
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocker failed-test legit-flake
    jpdjere
  2. Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/prerelease_packages·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA prerelease_packages "before each" hook for "should install latest stable version and ignore prerelease packages" #171428
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake
    jpdjere
  3. Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/install_latest_bundled_prebuilt_rules·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA install_bundled_prebuilt_rules should install prebuilt rules from the package that comes bundled with Kibana #171380
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake v8.12.0 v8.13.0
    jpdjere
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/basic/tests/add_prepackaged_rules·ts - detection engine api security and spaces enabled add_prepackaged_rules validation errors should give an error that the index must exist first if it does not exist before adding prepackaged rules #76832
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test

Rule Management
Preview Give feedback
  1. Failing Test: x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy·ts #171101
    8.13 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake v8.13.0
    nikitaindik
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index should report a mix of conflicts and a mix of successes #167919
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test

Misc
Preview Give feedback
No tasks being tracked yet.
@banderror banderror added test test-failure-flaky Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Dec 15, 2023
@banderror banderror self-assigned this Dec 15, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror mentioned this issue Dec 20, 2023
Open
@banderror banderror mentioned this issue Dec 20, 2023
Merged
@banderror
Copy link
Contributor Author

I triaged the tickets, added comments in some of them, and moved those of them that were legit flakes to #173731. We will take this ticket into work in January when people are back.

Also, I opened a PR that skips those tests that haven't been skipped yet: #173730.

banderror added a commit that referenced this issue Dec 20, 2023
@banderror
[Security Solution] Skip flaky Rule Management tests (#173730)
cdb1047 
**Resolves: #173469

## Summary

This PR:

- Skips flaky Rule Management tests that occurred recently and haven't
been already skipped by Kibana Operations.
- Adds `TODO` comments with links to the corresponding tickets.

We will be working on fixing and unskipping these and a few other tests
in January as part of #173731.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 20, 2023
@banderror
[Security Solution] Skip flaky Rule Management tests (elastic#173730)
8d86ee2 
**Resolves: elastic#173469

## Summary

This PR:

- Skips flaky Rule Management tests that occurred recently and haven't
been already skipped by Kibana Operations.
- Adds `TODO` comments with links to the corresponding tickets.

We will be working on fixing and unskipping these and a few other tests
in January as part of elastic#173731.

(cherry picked from commit cdb1047)
kibanamachine referenced this issue Dec 20, 2023
@kibanamachine @banderror
[8.12] [Security Solution] Skip flaky Rule Management tests (#173730) (…
cf47748 
…#173759)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Security Solution] Skip flaky Rule Management tests
(#173730)](#173730)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Georgii
Gorbachev","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-12-20T15:18:21Z","message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","test-failure-flaky","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","v8.12.0","v8.13.0"],"number":173730,"url":"https://github.com/elastic/kibana/pull/173730","mergeCommit":{"message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173730","number":173730,"mergeCommit":{"message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7"}}]}]
BACKPORT-->

Co-authored-by: Georgii Gorbachev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@banderror banderror

Labels
8.12 candidate Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test test-failure-flaky
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Security Solution] Skip flaky Rule Management tests banderror/kibana
2 participants
@banderror @elasticmachine