RMN integration test updates #15423
RMN integration test updates #15423
Conversation
dimkouv
commented
Nov 26, 2024
•
dimkouv
commented
- Re-configure and enable the disabled test cases in CI.
- Add cursing related integration tests (no issues found).
- Refactor the test, split testCase runner body into functions.
AER Report: CI Core ran successfully ✅AER Report: Operator UI CI ran successfully ✅ |
dimkouv
requested review from
cedric-cordenier,
winder,
rstout,
asoliman92,
makramkd,
mateusz-sekara,
0xAustinWang and
0xnogo
| @@ -75,7 +82,7 @@ func TestRMN_MultipleMessagesOnOneLaneNoWaitForExec(t *testing.T) { | |||
| func TestRMN_NotEnoughObservers(t *testing.T) { | |||
| runRmnTestCase(t, rmnTestCase{ | |||
| name: "one message but not enough observers, should not get a commit report", | |||
| passIfNoCommitAfter: time.Minute, // wait for a minute and assert that commit report was not delivered | |||
| passIfNoCommitAfter: 15 * time.Second, | |||
There was a problem hiding this comment.
Hmm, aren't we too aggressive with these durations? What is the commit is not delivered in 15 seconds, because something was lagging behind or the node didn't reach consensus, not because of the real reason?
I think that's a general problem with tests waiting for something "to not happen" within a time range. Is there any more reliable way of asserting that?
There was a problem hiding this comment.
That's one of my concerns as-well, I believe we do the same in other integration tests.
I couldn't figure out some better way. Any suggestions?
There was a problem hiding this comment.
This is a general issue with our testing framework that needs to be addressed at a higher level, we have tons of tests like this in smoke that are waiting to flake given ambiguous guestimates on timing. I agree with @mateusz-sekara and more generally I think smoke tests should test the integration of components/systems without executing under a "wait and see" model.
