Failing Version Bump: FTR Configs #23 / APM API tests trial no data fetching service anomalies with a trial license with ml jobs "before all" hook for "checks if alert is active"

[mistic]

APM API Integration tests (trial)
x-pack/test/apm_api_integration/tests/index.ts

APM API tests trial no data fetching service anomalies with a trial license with ml jobs "before all" hook for "checks if alert is active"

This failure is preventing the 8.9 version bump as well as the Elasticsearch promotion upon the branch cut.

• CI Run Failure
• Test Failure

Error: Expected status code 200, got 409 with body '{"error":{"root_cause":[{"type":"status_exception","reason":"No node found to start datafeed [apm-tx-metrics-production-datafeed], allocation explanation [cannot start datafeed [apm-tx-metrics-production-datafeed] because index [metrics-apm*,apm-*] does not exist, is closed, or is still initializing.]"}],"type":"status_exception","reason":"No node found to start datafeed [apm-tx-metrics-production-datafeed], allocation explanation [cannot start datafeed [apm-tx-metrics-production-datafeed] because index [metrics-apm*,apm-*] does not exist, is closed, or is still initializing.]"},"status":409}'
    at Assertion.assert (expect.js:100:11)
    at Assertion.eql (expect.js:244:8)
    at Object.assertResponseStatusCode (api.ts:85:31)
    at Object.startDatafeed (api.ts:839:12)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.createAndRunAnomalyDetectionLookbackJob (api.ts:869:7)
    at createAndRunApmMlJob (create_and_run_apm_ml_jobs.ts:39:3)
    at createAndRunApmMlJobs (create_and_run_apm_ml_jobs.ts:25:5)
    at Context.<anonymous> (anomaly_alert.spec.ts:77:11)
    at Object.apply (wrap_function.js:73:16) {
  actual: '409',
  expected: '200',
  showDiff: true
}

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[mistic]

Skipped.

main: e2ea83d

[achyutjhunjhunwala]

Running this test in Trial mode gives 403 instead of 409 currently. The createAndRunApmMlJobs function throws a 403 FORBIDDEN error when running in TRIAL mode. Need some one with more context to have a look at this issue

cc: @gbamparop

[sorenlouv]

@achyutjhunjhunwala Can you reach out to the ML team and hear them out?

[achyutjhunjhunwala]

Please ignore my previous comment, it was due to me running the server with basic flag and the runner with trial flag which caused the issue.

This skipped test genuinely has some issue. It simply gets stuck and times out every time you try to run it.

Culprit test - x-pack/test/apm_api_integration/tests/alerts/anomaly_alert.spec.ts

[achyutjhunjhunwala]

Closing this issue as original issue is no more happening and for the current issue, we have a reported bug - #160769