[CI] RecoveryTests#testRetentionPolicyChangeDuringRecovery failures

[jimczi]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux/2561

Does not reproduce locally with:

./gradlew -x :server:testScriptDocValuesMissingV6Behaviour \
  -Dtests.seed=7EC49556174B0437 \
  -Dtests.class=org.elasticsearch.indices.recovery.RecoveryTests \
  -Dtests.method="testRetentionPolicyChangeDuringRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=pt-PT \
  -Dtests.timezone=Australia/Eucla

but it failed 4 times last month in CI with the same exception:

Expected: <0>
     but: was <20>
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.indices.recovery.RecoveryTests.lambda$testRetentionPolicyChangeDuringRecovery$1(RecoveryTests.java:102)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
		... 39 more

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@dnhatn do you mind taking a look?

[droberts195]

The same error occurred today in a 6.3 build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=oraclelinux/213/console

FAILURE 10.3s J2 | RecoveryTests.testRetentionPolicyChangeDuringRecovery <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <0>
   >      but: was <20>
   > 	at __randomizedtesting.SeedInfo.seed([709FE84094F79193:EAA4166573FF31D7]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.indices.recovery.RecoveryTests.lambda$testRetentionPolicyChangeDuringRecovery$1(RecoveryTests.java:102)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:767)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:741)
   > 	at org.elasticsearch.indices.recovery.RecoveryTests.testRetentionPolicyChangeDuringRecovery(RecoveryTests.java:102)
   > 	at java.lang.Thread.run(Thread.java:748)

It didn't reproduce when I ran this against the 6.3 branch:

./gradlew :server:test \
  -Dtests.seed=709FE84094F79193 \
  -Dtests.class=org.elasticsearch.indices.recovery.RecoveryTests \
  -Dtests.method="testRetentionPolicyChangeDuringRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=ko \
  -Dtests.timezone=Asia/Irkutsk

[davidkyle]

Another instance that did not reproduce

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+release-tests/926/console

Expected: <0>
     but: was <20>
	at __randomizedtesting.SeedInfo.seed([A5DAAA599AC39A8A:3FE1547C7DCB3ACE]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.indices.recovery.RecoveryTests.lambda$testRetentionPolicyChangeDuringRecovery$1(RecoveryTests.java:102)

The failure actually came from a release build so I had to strip out -Dbuild.snapshot=false \ -Dtests.jvm.argline="-Dbuild.snapshot=false" from the reproduce line to get it to build. Still does not reproduce

./gradlew :server:test \
  -Dtests.seed=A5DAAA599AC39A8A \
  -Dtests.class=org.elasticsearch.indices.recovery.RecoveryTests \
  -Dtests.method="testRetentionPolicyChangeDuringRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=be \
  -Dtests.timezone=Indian/Mahe

[jkakavas]

This also failed today in 6.x https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/2276/console

REPRODUCE WITH: ./gradlew :server:test \
  -Dtests.seed=5D41AD8C90C233F3 \
  -Dtests.class=org.elasticsearch.indices.recovery.RecoveryTests \
  -Dtests.method="testRetentionPolicyChangeDuringRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=et-EE \
  -Dtests.timezone=Etc/GMT+0 \
  -Dcompiler.java=10 \
  -Druntime.java=8

07:14:01    > Expected: <0>
07:14:01    >      but: was <20>
07:14:01    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
07:14:01    > 		at org.elasticsearch.indices.recovery.RecoveryTests.lambda$testRetentionPolicyChangeDuringRecovery$1(RecoveryTests.java:106)
07:14:01    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:835)

[dnhatn]

I've muted this test as we should have enough information with the latest failure.

[dnhatn]

I opened #32967.