[CI] CardinalityWithRequestBreakerIT.testRequestBreaker fails on 7.9

[cbuescher]

Build scan:

https://gradle-enterprise.elastic.co/s/2coyhtowa22hi

Repro line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT.testRequestBreaker"
-Dtests.seed=ABEA4F928B9BAFCF
-Dtests.security.manager=true
-Dtests.locale=hr
-Dtests.timezone=Asia/Ust-Nera
-Druntime.java=8

Reproduces locally?:

yes

Applicable branches:

7.9

Failure history:

Another one on Sep 8th might be related: https://gradle-enterprise.elastic.co/s/e2ht4jwbic3dq

Failure excerpt:

java.lang.AssertionError: Request breaker not reset to 0 on node: node_s1
    Expected: <0L>
         but: was <524288L>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.test.InternalTestCluster.lambda$ensureEstimatedStats$39(InternalTestCluster.java:2501)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:951)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:924)
        at org.elasticsearch.test.InternalTestCluster.ensureEstimatedStats(InternalTestCluster.java:2499)
        at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:94)
        at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2523)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:598)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2233)

java.lang.RuntimeException: 2 arrays have not been released
        at org.elasticsearch.common.util.MockBigArrays.ensureAllArraysAreReleased(MockBigArrays.java:70)
        at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:528)
        at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:368)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[cbuescher]

@henningandersen since you seem to have authored this test, could you take a look?

[cbuescher]

Just found another one today: https://gradle-enterprise.elastic.co/s/2coyhtowa22hi

[dimitris-athanasiou]

I just hit this type of failure today in a CI for a PR: https://gradle-enterprise.elastic.co/s/yryepezu7wpfk

java.lang.AssertionError: Request breaker not reset to 0 on node: node_s1
Expected: <0L>
     but: was <32L>
	at __randomizedtesting.SeedInfo.seed([5B1C745938F188BC:E34E4919D7315AF6]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.test.InternalTestCluster.lambda$ensureEstimatedStats$41(InternalTestCluster.java:2542)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1019)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:992)
	at org.elasticsearch.test.InternalTestCluster.ensureEstimatedStats(InternalTestCluster.java:2540)
	at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:83)
	at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2564)
	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:596)
	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2255)
	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)

[henningandersen]

This also failed recently in a regular 7.x build:

https://gradle-enterprise.elastic.co/s/kbti45zr4cp5s

Build history

[henningandersen]

This test failed in a master build with a leak:

[2021-05-11T09:19:14,003][ERROR][o.e.t.LeakTracker        ] [node_s3] LEAK: resource was not cleaned up before it was garbage-collected.

It can be reproduced using:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT.testRequestBreaker" -Dtests.seed=D3BE36364C08A43A -Dtests.locale=mt-MT -Dtests.timezone=Asia/Baghdad -Druntime.java=16 -Dtests.fips.enabled=true

Fix is incoming.

[henningandersen]

The last test failure was against 7.12. It looks like this stopped occurring, at least I have not been able to reproduce in a loop or with the reproduce lines. Will wait a bit longer before closing this.

Notice that the previous leak and fix was only in 7.x/master.

[henningandersen]

This stopped occurring. The fix in #72966 is unlikely to have fixed the incident reported by Dimitris, but it is very likely that a fix of resource handling has been made in anything from transport to cardinality aggregations and I am therefore closing this for now.