Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Fix flaky tests: Rule Management team 8.13 #173731

Closed
13 tasks done
Tracked by #153633
banderror opened this issue Dec 20, 2023 · 5 comments
Closed
13 tasks done
Tracked by #153633

[Security Solution] Fix flaky tests: Rule Management team 8.13 #173731

banderror opened this issue Dec 20, 2023 · 5 comments
Assignees
@maximpn
Labels
8.13 candidate Meta Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration

Comments

@banderror
Copy link
Contributor

banderror commented Dec 20, 2023 •
edited
Loading

Epic: #153633

Summary

Fix and unskip flaky/failing tests that belong to the Rule Management team.

Our "inbox" of failed and skipped tests:

Sub-tasks

Prebuilt rules
Preview Give feedback
  1. Failing test: Detection Engine Serverless / Prebuilt Rules Management API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/management/fleet_integration·ts - Detection Engine API - Prebuilt Rules Management @ess @serverless @skipInQA install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage #172107
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocker failed-test legit-flake
    jpdjere
  2. Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/prerelease_packages·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA prerelease_packages "before each" hook for "should install latest stable version and ignore prerelease packages" #171428
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake
    jpdjere
  3. Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/install_latest_bundled_prebuilt_rules·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA install_bundled_prebuilt_rules should install prebuilt rules from the package that comes bundled with Kibana #171380
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake v8.12.0 v8.13.0
    jpdjere
  4. Failing test: Detection Engine ESS / Update Prebuilt Rules Package - API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - Detection Engine API - Update Prebuilt Rules Package @ess @serverless @skipInQA update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package #174715
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake
    jpdjere
  5. Failing test: Rules Management - Update Prebuilt Rules Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - Detection Engine API - Update Prebuilt Rules Package @ess @serverless @skipInQA update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package #175250
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  6. Failing test: Rules Management - Update Prebuilt Rules Integration Tests - Serverless Env - Complete License.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - Detection Engine API - Update Prebuilt Rules Package @ess @serverless @skipInQA update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package #175481
    8.13 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere

Rule management and monitoring
Preview Give feedback
  1. Failing Test: x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy·ts #171101
    8.13 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake v8.13.0
    nikitaindik
  2. Failing test: Rule Bulk Actions API Integration Tests - ESS - Rule bulk actions logic.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/rule_bulk_actions/perform_bulk_action_ess·ts - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic legacy investigation fields should export rules with legacy investigation_fields and transform legacy field in response #173804
    8.13 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test legit-flake
    nikitaindik
  3. Failing test: Rules Management - Rule Import And Export Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/import_connectors·ts - Rules Management - Rule import export API @ess @brokenInServerless @skipInQA import action connectors "before each" hook for "imports a rule with an action connector" #176836
    8.13 candidate Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    maximpn

Related integrations
Preview Give feedback
  1. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/related_integrations/related_integrations·cy·ts - Related integrations related Integrations Advanced Setting is disabled rule details should display the integrations in the definition section should display the integrations in the definition section #173307
    8.12 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test skipped-test
    jpdjere
  2. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/related_integrations/related_integrations·cy·ts - Related integrations related Integrations Advanced Setting is disabled rules management table should not display a badge with the installed integrations should not display a badge with the installed integrations #173306
    8.12 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  3. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/related_integrations/related_integrations·cy·ts - Related integrations integrations installed (AWS CloudFront (enabled), AWS CloudTrail (disabled), System (enabled)) rules management table "before each" hook for "should display a badge with the installed integrations" "before each" hook for "should display a badge with the installed integrations" #176027
    Feature:Related Integrations Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
  4. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/related_integrations/related_integrations·cy·ts - Related integrations related Integrations Advanced Setting is disabled rules management table "before each" hook for "should not display a badge with the installed integrations" "before each" hook for "should not display a badge with the installed integrations" #176028
    Feature:Related Integrations Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test

Misc
Preview Give feedback
No tasks being tracked yet.
@banderror banderror added test Meta test_ui_functional technical debt Improvement of the software architecture and operational architecture test-api-integration Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team 8.13 candidate labels Dec 20, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

This was referenced Dec 20, 2023
Open
Merged
Closed
banderror added a commit that referenced this issue Dec 20, 2023
@banderror
[Security Solution] Skip flaky Rule Management tests (#173730)
cdb1047 
**Resolves: #173469

## Summary

This PR:

- Skips flaky Rule Management tests that occurred recently and haven't
been already skipped by Kibana Operations.
- Adds `TODO` comments with links to the corresponding tickets.

We will be working on fixing and unskipping these and a few other tests
in January as part of #173731.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 20, 2023
@banderror
[Security Solution] Skip flaky Rule Management tests (elastic#173730)
8d86ee2 
**Resolves: elastic#173469

## Summary

This PR:

- Skips flaky Rule Management tests that occurred recently and haven't
been already skipped by Kibana Operations.
- Adds `TODO` comments with links to the corresponding tickets.

We will be working on fixing and unskipping these and a few other tests
in January as part of elastic#173731.

(cherry picked from commit cdb1047)
kibanamachine referenced this issue Dec 20, 2023
@kibanamachine @banderror
[8.12] [Security Solution] Skip flaky Rule Management tests (#173730) (…
cf47748 
…#173759)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Security Solution] Skip flaky Rule Management tests
(#173730)](#173730)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Georgii
Gorbachev","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-12-20T15:18:21Z","message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","test-failure-flaky","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","v8.12.0","v8.13.0"],"number":173730,"url":"https://github.com/elastic/kibana/pull/173730","mergeCommit":{"message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173730","number":173730,"mergeCommit":{"message":"[Security
Solution] Skip flaky Rule Management tests (#173730)\n\n**Resolves:
https://github.com/elastic/kibana/issues/173469**\r\n\r\n##
Summary\r\n\r\nThis PR:\r\n\r\n- Skips flaky Rule Management tests that
occurred recently and haven't\r\nbeen already skipped by Kibana
Operations.\r\n- Adds `TODO` comments with links to the corresponding
tickets.\r\n\r\nWe will be working on fixing and unskipping these and a
few other tests\r\nin January as part of
https://github.com/elastic/kibana/issues/173731.","sha":"cdb10478ce1270e8ce0ec37ddff98888e92f0af7"}}]}]
BACKPORT-->

Co-authored-by: Georgii Gorbachev <[email protected]>
@banderror
Copy link
Contributor Author

All the failed-test tickets have been addressed and closed 🕺 -- so closing this ticket as well.
We will reopen this one or create a new one for the 8.14 cycle once we get a new flaky test.
Thank you @jpdjere and @nikitaindik for your efforts ❤️

@banderror
Copy link
Contributor Author

Reopened #176836

@banderror banderror reopened this Feb 13, 2024
@banderror banderror assigned maximpn and unassigned jpdjere and nikitaindik Feb 13, 2024
@maximpn
Copy link
Contributor

maximpn commented Feb 19, 2024 •
edited
Loading

Reopened #176836

#176836 has been addressed and fixed. See the comment for more details #176836 (comment).

@maximpn maximpn closed this as completed Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

Labels
8.13 candidate Meta Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@maximpn @jpdjere @banderror @elasticmachine @nikitaindik