Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows·cy·ts - Detection rules, Prebuilt Rules Installation and Update workflow Installation of prebuilt rules package via Fleet should install package from Fleet in the background should install package from Fleet in the background #163586

Closed
Tracked by #161507
kibanamachine opened this issue Aug 10, 2023 · 3 comments · Fixed by #163468
Closed
Tracked by #161507

Failing test: Security Solution Cypress.x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows·cy·ts - Detection rules, Prebuilt Rules Installation and Update workflow Installation of prebuilt rules package via Fleet should install package from Fleet in the background should install package from Fleet in the background #163586

kibanamachine opened this issue Aug 10, 2023 · 3 comments · Fixed by #163468
Assignees
@jpdjere
Labels
8.10 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

A test failed on a tracked branch

TypeError: Cannot read properties of undefined (reading 'installSource')
    at eval (webpack:///./e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts:73:34)
    at Context.eval (webpack:///./e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts:70:50)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Aug 10, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Aug 10, 2023
@kibanamachine kibanamachine added the Team:Detection Rule Management Security Detection Rule Management Team label Aug 10, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Aug 10, 2023
@banderror banderror added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area 8.10 candidate labels Aug 10, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror mentioned this issue Aug 10, 2023
Closed
@banderror
Copy link
Contributor

@jpdjere This is the first automatically-created issue for our cypress tests, correctly labeled with the Team:Detection Rule Management label 🙌

Probably a duplicate of #163447, but I still added it to the Todo column and #161507

@jpdjere jpdjere mentioned this issue Aug 11, 2023
Merged
@jpdjere jpdjere linked a pull request Aug 11, 2023 that will close this issue
[Security Solution] Flaky Cypress test: Detection rules, Prebuilt Rules Installation and Update workflow - Installation of prebuilt rules package via Fleet should install package from Fleet in the background #163468
Merged
jpdjere added a commit that referenced this issue Aug 14, 2023
@jpdjere
[Security Solution] Flaky Cypress test: Detection rules, Prebuilt Rul…
aaefcdf 
…es Installation and Update workflow - Installation of prebuilt rules package via Fleet should install package from Fleet in the background (#163468)

Fixes: #163447
#163586

## Summary

- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts`
- Test title: `Detection rules, Prebuilt Rules Installation and Update
workflow - Installation of prebuilt rules package via Fleet should
install package from Fleet in the background`

## Details

- Initially ran the flaky test runner with multiple iterations and all
gave succesful results, i.e. no flakiness or failed tests.
- But: after checking the logs for the failed tests in the original
failed build, discovered that the reason the test failed is because:
- when checking Fleet's response for the installation of the
`security_detection_engine`, the API response was not as expected from
the API spec:
```
**Expected:** [{ name: 'security_detection_engine', installSource: 'registry' }]
**Actual:** [{ name: 'security_detection_engine', installSource: undefined }]
```
Since we cannot rely 100% that the Fleet API will return the correct
value for the installSource, this PR deletes this part of the test to
prevent any type of flakiness caused by external factors such as this.
@jpdjere jpdjere mentioned this issue Aug 23, 2023
Merged
jpdjere added a commit that referenced this issue Aug 24, 2023
@jpdjere
[8.9] [Security Solution] Flaky Cypress test: Detection rules, Prebui…
7b2b4a4 
…lt Rules Installation and Update workflow - Installation of prebuilt rules package via Fleet should install package from Fleet in the background (#164566)

**NOTE: This is a manual backport of
#163468 to 8.9.**

**Original PR description:**

Fixes: #163447
Fixes: #163586

## Summary

- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.cy.ts`
- Test title: `Detection rules, Prebuilt Rules Installation and Update
workflow - Installation of prebuilt rules package via Fleet should
install package from Fleet in the background`

## Details

- Initially ran the flaky test runner with multiple iterations and all
gave succesful results, i.e. no flakiness or failed tests.
- But: after checking the logs for the failed tests in the original
failed build, discovered that the reason the test failed is because:
- when checking Fleet's response for the installation of the
`security_detection_engine`, the API response was not as expected from
the API spec:
```
**Expected:** [{ name: 'security_detection_engine', installSource: 'registry' }]
**Actual:** [{ name: 'security_detection_engine', installSource: undefined }]
```
Since we cannot rely 100% that the Fleet API will return the correct
value for the installSource, this PR deletes this part of the test to
prevent any type of flakiness caused by external factors such as this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jpdjere jpdjere

Labels
8.10 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
4 participants
@jpdjere @banderror @elasticmachine @kibanamachine