[test_serverless/api_integration] convert alerting_wait_for_helpers into FTR service

[dmlemeshko]

Summary

This PR converts helpers into FTR services that should provide better test capabilities and improve stability/error output.

Before change:

 └-> should be active
         └-> "before each" hook: global before each for "should be active"
         └- ✖ fail: serverless observability API Threshold rule - AVG - PCT - FIRED Rule creation should be active
         │      Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/dmle/github/kibana/x-pack/test_serverless/api_integration/test_suites/observability/threshold_rule/avg_pct_fired.ts)
         │       at listOnTimeout (node:internal/timers:559:17)
         │       at processTimers (node:internal/timers:502:7)

The test is not handling its state properly and fails on mocha timeout (6 minutes) without proper error message. It leads the test suite usually taking ~5 min keep running for over 1 hour

After change:

2)    serverless observability API
 │       Threshold rule - AVG - PCT - FIRED
 │         Rule creation
 │           should be active:
 │
 │      Error: 'ruleId' is undefined
 │       at Object.waitForRuleStatus (alerting_api.ts:31:15)
 │       at Context.apply (avg_pct_fired.ts:122:51)
 │       at Object.apply (wrap_function.js:73:30)

Since the follow-up tests depend on the first test to success, we can either move Rule creation to before hook or check for ruleId to be defined. If it is defined, retry service will pull for status up to 90 sec (important to keep below Mocha timeout so you can get error) and fail if it is not matching expected one.

[fkanout]

AO changes LGTM. Thank you @dmlemeshko!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: c9e1837

Metrics [docs]

✅ unchanged

History

• 💚 Build #146518 succeeded 6272d79
• 💚 Build #145246 succeeded 73525e5
• 💔 Build #145183 failed 4647e38

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream