Stale events 53411

[cro]

What does this PR do?

Adds event_listen_queue_max_seconds to fix #53411

What issues does this PR fix or reference?

#53411

Previous Behavior

If event_listen_queue is set to large values or if it is set but the event bus is not busy enough, events can languish in the queue and cause pathological behavior (CLI reporting Minion did not return, job results taking too long to return.

New Behavior

The event_listen_queue will be flushed if the oldest event in the queue is older than event_listen_queue_max_seconds regardless of the number of events in the queue.

Commits signed with GPG?

Yes

[dwoz]

Looks good but we are waiting on some tests for this change.

[dmurphy18]

The extraneous conditional
else:
too_long_in_queue = False

might be unneeded, since line 1272 resets to False, unless I missed the need for it, in which case let me know what I missed.
otherwise looks good,

[cro]

After banging my head against this for some time I think we will need to resort to a manual test.

1. The test suite currently starts a master but never restarts one so unless we make this config the default there's no way to alter the configuration
2. There are no tests (that I can find) in the suite that verify functionality of non-default master config options.

If there is an approach I am unaware of I'm happy to explore it.

[waynew]

Both of those points sound accurate to me.

I would love it if we had an automated suite that roughly does what we do with the current integration suite, but instead was more like just running a salt orchestration, and we verified that each of the state result was what we expected. Basically an automated manual test suite 😂

[cro]

K, fixed the doc test failure I think.

[cro]

And rebased against 2018.3. I think this will be ready for merge if all the tests pass again.

[waynew]

@cro I think we also need some tests, or is this fixing some existing tests?

[cro]

I would like to write tests, but according to my understanding of the test suite as it stands:

After banging my head against this for some time I think we will need to resort to a manual test.

1. The test suite currently starts a master but never restarts one so unless we make this config the default there's no way to alter the configuration
2. There are no tests (that I can find) in the suite that verify functionality of non-default master config options.

If there is an approach I am unaware of I'm happy to explore it.

[thatch45]

I approve this merge as an exception to the tests writing rule.

[cro]

Re-run all