[ResponseOps][Task Manager][mget Claimer] finish jest, functional, integration tests

[pmuellr]

In PR implement task claiming strategy mget #180485 we implemented an alternative task claiming strategy, but it has the following problem:

The jest and function tests are incomplete.

The jest tests do not test all the conditions of the default claimer, which they should.

Some of the function tests added in #180485 were skipped as they were flaky or always failing - those obviously need to be fixed. The functional tests that are there were copied from the existing task manager tests in https://github.com/elastic/kibana/tree/main/x-pack/test/plugin_api_integration/test_suites/task_manager - we should see if there are other function tests we can copy.

We should also see if we can just reuse the existing function tests drectly, instead of copying them. We'll still need a separate config, but maybe we can at least remove the copies of the test modules.

We should also see if we can build a simple integration test that launches multiple Kibanas, and ensures the claimers are working as expected (unclear what that means). We should actually try to make this test work with both claimers!

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

@pmuellr when we switch the default task claimer to mget, and say we make the mget specific test suite now update_by_query specific (for legacy purposes), does this reduce or eliminate the list of tests we need to add?

I'm working on a draft PR to see what test failures will come up once we switch to mget (and a poll interval of 500) => #190148

[pmuellr]

My initial thought was ...

We should also see if we can just reuse the existing function tests drectly, instead of copying them. We'll still need a separate config, but maybe we can at least remove the copies of the test modules.

So likely the FT tests, which aren't sensitive to the claimer, should hopefully be reused - but we need to figure out how to do that :-). FTR work, but not new tests per-se.

[mikecote]

Gotcha, so if we switch the claimer used for all the tests to mget, we would get all the coverage necessary? Leaving behind update_by_query that now wouldn't be as fully covered. Perhaps that's sufficient? 🤔

[pmuellr]

Gotcha, so if we switch the claimer used for all the tests to mget, we would get all the coverage necessary? Leaving behind update_by_query that now wouldn't be as fully covered. Perhaps that's sufficient? 🤔

Seems like to me it's the minimum anyway. We will likely want some specialized tests, if possible, to deal with multiple costs, cost left over after claim, etc, etc - that's different than UBQ claimer. But it would at least be good to see all the existing tests, including implicitly the ones for alerting rules, running with the new claimer, just to make sure we haven't broken anything we already have tests for.

Perhaps we should create a floating PR that just whacks all the FTR configs to use the new claimer, and see what happens in CI. We can rebase it when we want to test any changes, throw it away when we decide what to REALLY do to test both of the claimers ...

[mikecote]

I have this PR #190148 that hits all the test suites using the mget task claimer. There was a few tests I had to modify to fix race conditions but the results are looking good so far!