Failing test: Serverless Observability Feature Flags API Integration Tests.x-pack/test_serverless/api_integration/test_suites/observability/custom_threshold_rule/group_by_fired·ts - Serverless observability API - feature flags Custom Threshold Rule Custom Threshold rule - GROUP_BY - FIRED Rule creation should set correct action variables

[kibanamachine]

A test failed on a tracked branch

Error: expected undefined to sort of equal 'observability.rules.custom_threshold'
    at Assertion.assert (expect.js:100:11)
    at Assertion.eql (expect.js:244:8)
    at Context.<anonymous> (group_by_fired.ts:253:53)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  actual: undefined,
  expected: 'observability.rules.custom_threshold',
  showDiff: true
}

First failure: CI Build - main

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[maryam-saeidi]

History of this failure

From the 4 instances above, 2 of them were wrong values in test that was fixed in the PR, and for the other 2, the root cause mentioned in the logs is:

index_not_found_exception: no such index [alert-action-threshold]

In both cases, we have the following error log in task manager:

[00:02:02]             │ proc [kibana] [2024-01-29T10:53:05.557+00:00][ERROR][plugins.taskManager] Task actions:.index "7484ee48-6ac2-4c5c-a778-8fef443c4025" 
failed: Error: Saved object [action/6a53161a-9e45-450e-b77a-bc3bfa399b96] not found 
{"tags":["actions:.index","7484ee48-6ac2-4c5c-a778-8fef443c4025","task-run-failed"],
"error":{"stack_trace":"Error: Saved object [action/6a53161a-9e45-450e-b77a-bc3bfa399b96] not found\n    
at Function.createGenericNotFoundError (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-server/src/saved_objects_error_helpers.js:236:37)\n    at performGet (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-api-server-internal/src/lib/apis/get.js:73:60)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at SavedObjectsRepository.get (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-api-server-internal/src/lib/repository.js:218:12)\n    at Object.getDecryptedAsInternalUser (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/encrypted-saved-objects-plugin/server/saved_objects/index.js:50:29)\n    at ActionExecutor.getActionInfoInternal (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/actions-plugin/server/lib/action_executor.js:377:25)\n    at Object.loadIndirectParams (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/actions-plugin/server/lib/task_runner_factory.js:81:30)\n    at TaskManagerRunner.validateIndirectTaskParams (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:419:11)\n    at TaskManagerRunner.run (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:320:34)"},"service":{"node":{"roles":["background_tasks","ui"]}}}

Started a discussion in Slack.

Update

It seems eventually a document is found but the information in that document is not as we expetected:

debg Found 1 docs, looking for atleast 1.

[maryam-saeidi]

I made some improvements in this PR and will close this issue as I didn't see a failure when I ran the test 200 times after the improvement. I've also added some logs, so if it happens again, hopefully, it will be easier to investigate it.

[kibanamachine]

New failure: CI Build - 8.14