[TEST] NoMasterNodeIT.testNoMasterActions fails

[ywelsch]

This test sporadically fails when it runs on a CI machine that slows down a bit. The test relies on timing, which is a big NONO:

long now = System.currentTimeMillis();
try {
    builder.get();
    fail("Expected ClusterBlockException");
} catch (ClusterBlockException e) {
    if (indexShouldBeAutoCreated) {
        // timeout is 200
        assertThat(System.currentTimeMillis() - now, greaterThan(timeout.millis() - 50));
        assertThat(e.status(), equalTo(RestStatus.SERVICE_UNAVAILABLE));
    } else {
        // timeout is 5000
        assertThat(System.currentTimeMillis() - now, lessThan(timeout.millis() + 300));
    }
}

Failure:

ERROR   33.8s J0 | NoMasterNodeIT.testNoMasterActions <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: a value less than <250L>
   >      but: <269L> was greater than <250L>
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.cluster.NoMasterNodeIT.checkWriteAction(NoMasterNodeIT.java:182)
   > 	at org.elasticsearch.cluster.NoMasterNodeIT.testNoMasterActions(NoMasterNodeIT.java:127)
   > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: MasterNotDiscoveredException[null]
   > 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:209)
...

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.4+multijob-windows-compatibility/46/consoleFull

[javanna]

another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-windows-compatibility/85/console .

[javanna]

This has been failing more frequently over the last couple of days:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-windows-compatibility/82/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-windows-compatibility/77/console

Suspicious that all of these failures on Windows and 5.x.

[abeyad]

Another failure on Windows, 5.x: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-windows-compatibility/129/console

Reassigning from Areek to @bleskes for now. @bleskes feel free to reassign to someone else as appropriate

[jasontedor]

This test is still failing. Can you take a look @bleskes?

Here are logs: logs.zip.

[spinscale]

@bleskes this failed on 5.3 as well - might be worth a backport?

CI link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-windows-compatibility/72/console

[jasontedor]

The 5.3 branch should be set to push only in CI and we should then not worry about backporting fixes for sporadic test failures.