-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/prerelease_packages·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA prerelease_packages "before each" hook for "should install latest stable version and ignore prerelease packages" #171428
Comments
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
This seems to be a legit flake, but could be hard to reproduce. The error happened in this function: /**
* Remove all prebuilt rule assets from the security solution savedObjects index
* @param es The ElasticSearch handle
*/
export const deleteAllPrebuiltRuleAssets = async (es: Client): Promise<void> => {
await es.deleteByQuery({
index: SECURITY_SOLUTION_SAVED_OBJECT_INDEX,
q: 'type:security-rule',
wait_for_completion: true,
refresh: true,
body: {},
});
}; The error was:
Could Fleet or our code in the app be installing or updating |
I'm not skipping this test because:
|
@MadameSheema Hi Glo, maybe you can help us elucidate this failing test. What failed in this particular test suite is the It failed here with a However, since this util is called only once in the whole test file , I would like to understand which and how many test suites are run in parallel in these FTR Integration tests. For this specific directory, and their associated config file, we have only two test suites: So a couple of questions:
EDIT: I could only reproduce this issue in Serverless (1 in 500 runs), maybe the env it runs on is related? |
… Integration tests (#174185) ## Summary Fixes: #171428 **NOTE: the test where this was reported wasn't skipped, so this PR does not unskip any tests.** However, the Flaky Test Runs help us determine that the issue is no longer reproducible. The `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`, presumably from `security-rule` assets that were attempted to be deleted while they were being updated by a parallel process. This PR wraps the `es.deleteByQuery` calls in the utils `deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new `retryIfConflict` helper, that will retry the operation if the ES request fails with a `409`. ## Flaky test run `bundled_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790 `large_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791 `update_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792 `management` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793 ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
… Integration tests (elastic#174185) ## Summary Fixes: elastic#171428 **NOTE: the test where this was reported wasn't skipped, so this PR does not unskip any tests.** However, the Flaky Test Runs help us determine that the issue is no longer reproducible. The `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`, presumably from `security-rule` assets that were attempted to be deleted while they were being updated by a parallel process. This PR wraps the `es.deleteByQuery` calls in the utils `deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new `retryIfConflict` helper, that will retry the operation if the ES request fails with a `409`. ## Flaky test run `bundled_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790 `large_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791 `update_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792 `management` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793 ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit b8c7306) # Conflicts: # x-pack/test/security_solution_api_integration/package.json
…icts in Integration tests (#174185) (#174762) # Backport This will backport the following commits from `main` to `8.12`: - [[Security Solution] Add `retryIfConflict` util for `409` conflicts in Integration tests (#174185)](#174185) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Juan Pablo Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2024-01-11T12:39:45Z","message":"[Security Solution] Add `retryIfConflict` util for `409` conflicts in Integration tests (#174185)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/171428\r\n\r\n**NOTE: the test where this was reported wasn't skipped, so this PR does\r\nnot unskip any tests.** However, the Flaky Test Runs help us determine\r\nthat the issue is no longer reproducible.\r\n\r\nThe `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`,\r\npresumably from `security-rule` assets that were attempted to be deleted\r\nwhile they were being updated by a parallel process.\r\n\r\nThis PR wraps the `es.deleteByQuery` calls in the utils\r\n`deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new\r\n`retryIfConflict` helper, that will retry the operation if the ES\r\nrequest fails with a `409`.\r\n\r\n## Flaky test run\r\n\r\n`bundled_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790\r\n\r\n`large_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791\r\n\r\n`update_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792\r\n\r\n`management` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b8c7306d241807b68bedbd477dcec232e203f6ad","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","v8.12.0","v8.12.1","v8.13.0"],"number":174185,"url":"https://github.com/elastic/kibana/pull/174185","mergeCommit":{"message":"[Security Solution] Add `retryIfConflict` util for `409` conflicts in Integration tests (#174185)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/171428\r\n\r\n**NOTE: the test where this was reported wasn't skipped, so this PR does\r\nnot unskip any tests.** However, the Flaky Test Runs help us determine\r\nthat the issue is no longer reproducible.\r\n\r\nThe `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`,\r\npresumably from `security-rule` assets that were attempted to be deleted\r\nwhile they were being updated by a parallel process.\r\n\r\nThis PR wraps the `es.deleteByQuery` calls in the utils\r\n`deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new\r\n`retryIfConflict` helper, that will retry the operation if the ES\r\nrequest fails with a `409`.\r\n\r\n## Flaky test run\r\n\r\n`bundled_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790\r\n\r\n`large_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791\r\n\r\n`update_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792\r\n\r\n`management` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b8c7306d241807b68bedbd477dcec232e203f6ad"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/174185","number":174185,"mergeCommit":{"message":"[Security Solution] Add `retryIfConflict` util for `409` conflicts in Integration tests (#174185)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/171428\r\n\r\n**NOTE: the test where this was reported wasn't skipped, so this PR does\r\nnot unskip any tests.** However, the Flaky Test Runs help us determine\r\nthat the issue is no longer reproducible.\r\n\r\nThe `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`,\r\npresumably from `security-rule` assets that were attempted to be deleted\r\nwhile they were being updated by a parallel process.\r\n\r\nThis PR wraps the `es.deleteByQuery` calls in the utils\r\n`deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new\r\n`retryIfConflict` helper, that will retry the operation if the ES\r\nrequest fails with a `409`.\r\n\r\n## Flaky test run\r\n\r\n`bundled_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790\r\n\r\n`large_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791\r\n\r\n`update_prebuilt_rules_package` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792\r\n\r\n`management` - **ESS** and **Serverless**:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b8c7306d241807b68bedbd477dcec232e203f6ad"}}]}] BACKPORT-->
A test failed on a tracked branch
First failure: CI Build - main
The text was updated successfully, but these errors were encountered: