[8.9] [Security Solution] Fix flaky test: x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts - update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package

[jpdjere]

NOTE: This is a manual backport of #163241. It had to be done manually because of changes that happened from 8.9 -> 8.10 like the common folder restructuring and some type changes because of the OpenAPI effort, but it is mostly the changes to the same files as that PR.

Original PR description follows:

Fixes: #162658

Summary

• Fixes flaky test: x-pack/test/detection_engine_api_integration/security_and_spaces/update_prebuilt_rules_package/update_prebuilt_rules_package·ts
• Test title: update_prebuilt_rules_package should allow user to install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package

Passing flaky test runner

300 runs: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2799

Root cause and what this PR does

• On initial investigation, the flaky test runner was executed for this test with 300 iterations and all of them succeeded. This gives us great confidence that the test is not actually flaky.
• Further investigation showed that @kibanamachine reported this tests as failing when running the CI for two backport PRs to 8.9:
  • https://buildkite.com/elastic/kibana-on-merge/builds/33282#0189987d-3a80-49c2-8332-3105ec3c2109
  • https://buildkite.com/elastic/kibana-on-merge/builds/33444#0189b1fa-4bc4-4422-9ce9-5c9a24f11ad5
• These flakiness was caused by a race condition between the writing of rules into indeces, and them being made available for reading. This is a known issue, already and solved for other integration test, by manually refreshing ES's indeces.
• In order to reduce the probability of flakiness in a similar scenario, this PR adds code to refresh the indices after each rule installation or upgrade during the test.

Refactor

• Moves the refreshing of the indexes within the utility function that write to indexes:
  • installPrebuiltRules
  • upgradePrebuiltRules
  • installPrebuiltRulesAndTimelines (legacy, but still used)
  • installPrebuiltRulesFleetPackage
• Creates 2 new utils:
  • installPrebuiltRulesPackageByVersion, which installs security_detection_engine package via Fleet API, with its version passed in as param
  • getInstalledRules, reusable function to fetch all installed rules

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 1d61412

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	13	15	+2
securitySolution	416	420	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	14	16	+2
securitySolution	497	501	+4
total			+6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere