-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Add
retryIfConflict util for 409 conflicts in…
… Integration tests (#174185) ## Summary Fixes: #171428 **NOTE: the test where this was reported wasn't skipped, so this PR does not unskip any tests.** However, the Flaky Test Runs help us determine that the issue is no longer reproducible. The `deleteAllPrebuiltRuleAssets` utility reported a `409 Conflict`, presumably from `security-rule` assets that were attempted to be deleted while they were being updated by a parallel process. This PR wraps the `es.deleteByQuery` calls in the utils `deleteAllPrebuiltRuleAssets` and `deleteAllTimelines` with a new `retryIfConflict` helper, that will retry the operation if the ES request fails with a `409`. ## Flaky test run `bundled_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4790 `large_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4791 `update_prebuilt_rules_package` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4792 `management` - **ESS** and **Serverless**: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4793 ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- Loading branch information
Showing
13 changed files
with
105 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
53 changes: 53 additions & 0 deletions
53
..._api_integration/test_suites/detections_response/utils/retry_delete_by_query_conflicts.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { DeleteByQueryResponse } from '@elastic/elasticsearch/lib/api/types'; | ||
| import { ToolingLog } from '@kbn/tooling-log'; | ||
|
|
||
| // Number of times to retry when conflicts occur | ||
| const RETRY_ATTEMPTS = 2; | ||
|
|
||
| // Delay between retries when conflicts occur | ||
| const RETRY_DELAY = 200; | ||
|
|
||
| /* | ||
| * Retry an Elasticsearch deleteByQuery operation if it runs into 409 Conflicts, | ||
| * up to a maximum number of attempts. | ||
| */ | ||
| export async function retryIfDeleteByQueryConflicts<T>( | ||
| logger: ToolingLog, | ||
| name: string, | ||
| operation: () => Promise<DeleteByQueryResponse>, | ||
| retries: number = RETRY_ATTEMPTS, | ||
| retryDelay: number = RETRY_DELAY | ||
| ): Promise<DeleteByQueryResponse> { | ||
| const operationResult = await operation(); | ||
| if (!operationResult.failures || operationResult.failures?.length === 0) { | ||
| return operationResult; | ||
| } | ||
|
|
||
| for (const failure of operationResult.failures) { | ||
| if (failure.status === 409) { | ||
| // if no retries left, throw it | ||
| if (retries <= 0) { | ||
| logger.error(`${name} conflict, exceeded retries`); | ||
| throw new Error(`${name} conflict, exceeded retries`); | ||
| } | ||
|
|
||
| // Otherwise, delay a bit before retrying | ||
| logger.debug(`${name} conflict, retrying ...`); | ||
| await waitBeforeNextRetry(retryDelay); | ||
| return await retryIfDeleteByQueryConflicts(logger, name, operation, retries - 1); | ||
| } | ||
| } | ||
|
|
||
| return operationResult; | ||
| } | ||
|
|
||
| async function waitBeforeNextRetry(retryDelay: number): Promise<void> { | ||
| await new Promise((resolve) => setTimeout(resolve, retryDelay)); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters