Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/spaces_only/tests/alerting/event_log·ts - alerting api integration spaces only Alerting eventLog should generate expected events for normal operation

[kibanamachine]

A test failed on a tracked branch

Error: expected false to equal true
    at Assertion.assert (/dev/shm/workspace/parallel/5/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/parallel/5/kibana/packages/kbn-expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/parallel/5/kibana/packages/kbn-expect/expect.js:69:22)
    at Context.it (test/alerting_api_integration/spaces_only/tests/alerting/event_log.ts:91:44)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[mistic]

That test as failed again both on master and 7.x and it looks flaky, skipping for now.

master: 0a0672f
7.x: 775f543

[pmuellr]

The test failure is on line 91 here:

kibana/x-pack/test/alerting_api_integration/spaces_only/tests/alerting/event_log.ts

Lines 92 to 95 in 31a5b15

	expect(executeEvents.length >= 4).to.be(true);
	expect(executeActionEvents.length).to.be(2);
	expect(newInstanceEvents.length).to.be(1);
	expect(resolvedInstanceEvents.length).to.be(1);

It seems like we could change this to the following:

      expect(executeEvents.length >= 3).to.be(true);

And should probably use the correct expect usage of .not.lessThan(3) instead of the inlined boolean, to get a better diagnostic the next time it fails :-)

Rationale for changing:

• we only bother checking 3 execute events below this
• we only bother checking 1 execute action events below this
• for any cases where we checking a length to be 1 (or greater), that's already assumed in the retry() block above this, which will stop retrying once you have at least one of every action requested

It could be that the timing on this stuff changed once we started buffering the events. Eg, we now write out execute AFTER all the other events being checked, but back-dated to when the alert executor started. We used to write it out first. So missing an executeEvent makes some sense to me - it have been pushed out to a different buffer than the others, and so the last one might be "dropped" (but would magically appear a second later!).

[gmmorris]

It appeared to be a timing thing, as it did actually have 4 execute events.
You can see my solution here: #84189
It ensure we wait until the 4th even before exiting the retry....