Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package #162658

Closed
Tracked by #161507
kibanamachine opened this issue Jul 27, 2023 · 5 comments · Fixed by #163241
Closed
Tracked by #161507

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package #162658

kibanamachine opened this issue Jul 27, 2023 · 5 comments · Fixed by #163241
Assignees
@jpdjere
Labels
8.10 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Jul 27, 2023 •
edited
Loading

A test failed on a tracked branch

JestAssertionError: expect(received).toBe(expected) // Object.is equality

Expected: 276
Received: 756
    at Context.<anonymous> (update_prebuilt_rules_package.ts:136:62)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (wrap_function.js:73:16) {
  matcherResult: {
    actual: 756,
    expected: 276,
    message: '\x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoBe\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m) // Object.is equality\x1B[22m\n' +
      '\n' +
      'Expected: \x1B[32m276\x1B[39m\n' +
      'Received: \x1B[31m756\x1B[39m',
    name: 'toBe',
    pass: false
  }
}

First failure: CI Build - 8.9
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jul 27, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Jul 27, 2023
@kibanamachine kibanamachine added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Jul 27, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jul 27, 2023
@MadameSheema MadameSheema added triage_needed Team:Detections and Resp Security Detection Response Team Team:Detection Engine Security Solution Detection Engine Area labels Jul 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror banderror added Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area 8.10 candidate and removed Team:Detection Engine Security Solution Detection Engine Area labels Jul 31, 2023
@banderror banderror mentioned this issue Jul 31, 2023
Closed
@banderror banderror assigned jpdjere and unassigned banderror Jul 31, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@jpdjere
Copy link
Contributor

jpdjere commented Aug 7, 2023

Flaky test run: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2795

@jpdjere
Copy link
Contributor

jpdjere commented Aug 7, 2023

Second attempt, with 300 runs: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2799

@jpdjere jpdjere mentioned this issue Aug 8, 2023
Merged
jpdjere added a commit that referenced this issue Aug 11, 2023
@jpdjere @kibanamachine
[Security Solution] Fix flaky test: x-pack/test/detection_engine_api_…
2ba659d 
…integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package (#163241)

Fixes: #162658

## Summary

- Fixes flaky test:
`x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts`
- Test title: `update_prebuilt_rules_package should allow user to
install prebuilt rules from scratch, then install new rules and upgrade
existing rules from the new package`

## Passing flaky test runner

300 runs:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2799

## Root cause and what this PR does

- On initial investigation, the flaky test runner was executed for this
test with 300 iterations and all of them succeeded. This gives us great
confidence that the test is not actually flaky.
- Further investigation showed that @kibanamachine reported this tests
as failing when running the CI for two backport PRs to `8.9`:
-
https://buildkite.com/elastic/kibana-on-merge/builds/33282#0189987d-3a80-49c2-8332-3105ec3c2109
-
https://buildkite.com/elastic/kibana-on-merge/builds/33444#0189b1fa-4bc4-4422-9ce9-5c9a24f11ad5
- These flakiness was caused **by a race condition** between the writing
of rules into indeces, and them being made available for reading. This
is a known issue, already and solved for other integration test, by
manually refreshing ES's indeces.
- In order to reduce the probability of flakiness in a similar scenario,
this PR adds code to refresh the indices after each rule installation or
upgrade during the test.

## Refactor

- Moves the refreshing of the indexes within the utility function that
write to indexes:
    - `installPrebuiltRules`
    - `upgradePrebuiltRules`
    - `installPrebuiltRulesAndTimelines` (legacy, but still used)
    - `installPrebuiltRulesFleetPackage`
- Creates 2 new utils:
- `installPrebuiltRulesPackageByVersion`, which installs
`security_detection_engine` package via Fleet API, with its version
passed in as param
- `getInstalledRules`, reusable function to fetch all installed rules
    -

---------

Co-authored-by: kibanamachine <[email protected]>
@jpdjere jpdjere mentioned this issue Aug 23, 2023
Merged
jpdjere added a commit that referenced this issue Aug 24, 2023
@jpdjere
[8.9] [Security Solution] Fix flaky test: x-pack/test/detection_engin…
949e663 
…e_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package (#164558)

**NOTE: This is a manual backport of
#163241. It had to be done
manually because of changes that happened from 8.9 -> 8.10 like the
common folder restructuring and some type changes because of the OpenAPI
effort, but it is mostly the changes to the same files as that PR.**

**Original PR description follows:**

Fixes: #162658

## Summary

- Fixes flaky test:
`x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts`
- Test title: `update_prebuilt_rules_package should allow user to
install prebuilt rules from scratch, then install new rules and upgrade
existing rules from the new package`

## Passing flaky test runner

300 runs:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2799

## Root cause and what this PR does

- On initial investigation, the flaky test runner was executed for this
test with 300 iterations and all of them succeeded. This gives us great
confidence that the test is not actually flaky.
- Further investigation showed that @kibanamachine reported this tests
as failing when running the CI for two backport PRs to `8.9`:
-
https://buildkite.com/elastic/kibana-on-merge/builds/33282#0189987d-3a80-49c2-8332-3105ec3c2109
-
https://buildkite.com/elastic/kibana-on-merge/builds/33444#0189b1fa-4bc4-4422-9ce9-5c9a24f11ad5
- These flakiness was caused **by a race condition** between the writing
of rules into indeces, and them being made available for reading. This
is a known issue, already and solved for other integration test, by
manually refreshing ES's indeces.
- In order to reduce the probability of flakiness in a similar scenario,
this PR adds code to refresh the indices after each rule installation or
upgrade during the test.

## Refactor

- Moves the refreshing of the indexes within the utility function that
write to indexes:
    - `installPrebuiltRules`
    - `upgradePrebuiltRules`
    - `installPrebuiltRulesAndTimelines` (legacy, but still used)
    - `installPrebuiltRulesFleetPackage`
- Creates 2 new utils:
- `installPrebuiltRulesPackageByVersion`, which installs
`security_detection_engine` package via Fleet API, with its version
passed in as param
- `getInstalledRules`, reusable function to fetch all installed rules
    -
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jpdjere jpdjere

Labels
8.10 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
5 participants
@jpdjere @banderror @elasticmachine @MadameSheema @kibanamachine