RetentionLeaseIT fails in CI with NoSuchFileException on 6.8

[jaymode]

On the 6.8 branch, two tests of RetentionLeaseIT have failed in the past month. The nested exception in both cases was a NoSuchFileException, which leads me to believe that they may have failed for the same reason.

Build scans:
https://gradle-enterprise.elastic.co/s/g55d6mbwbohks
https://gradle-enterprise.elastic.co/s/jp5k4fcqo54x6

testCanRenewRetentionLeaseWithoutWaitingForShards failure:

REPRODUCE WITH: ./gradlew ':server:integTest' \
  -Dtests.seed=A0BC49BF06CE098F \
  -Dtests.class=org.elasticsearch.index.seqno.RetentionLeaseIT \
  -Dtests.method="testCanRenewRetentionLeaseWithoutWaitingForShards" \
  -Dtests.security.manager=true \
  -Dtests.locale=ca-FR \
  -Dtests.timezone=America/Metlakatla \
  -Dcompiler.java=12 \
  -Druntime.java=13

java.lang.AssertionError: ElasticsearchException[java.io.IOException: failed to read [id:1, file:/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/openjdk14/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_A0BC49BF06CE098F-001/tempDir-009/data/nodes/0/indices/EbjaG1FvQjO7gf6an7a1aQ/0/_state/retention-leases-1.st]]; nested: IOException[failed to read [id:1, file:/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/openjdk14/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_A0BC49BF06CE098F-001/tempDir-009/data/nodes/0/indices/EbjaG1FvQjO7gf6an7a1aQ/0/_state/retention-leases-1.st]]; nested: NoSuchFileException[/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/openjdk14/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_A0BC49BF06CE098F-001/tempDir-009/data/nodes/0/indices/EbjaG1FvQjO7gf6an7a1aQ/0/_state/retention-leases-1.st];
	at __randomizedtesting.SeedInfo.seed([A0BC49BF06CE098F:BA7DD8BC5C5C9EAD]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.lambda$testCanRenewRetentionLeaseWithoutWaitingForShards$37(RetentionLeaseIT.java:601)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.runWaitForShardsTest(RetentionLeaseIT.java:674)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.testCanRenewRetentionLeaseWithoutWaitingForShards(RetentionLeaseIT.java:583)

testCanRenewRetentionLeaseUnderBlock failure:

REPRODUCE WITH: ./gradlew ':server:integTest' \
  -Dtests.seed=863577CB03E7FAE8 \
  -Dtests.class=org.elasticsearch.index.seqno.RetentionLeaseIT \
  -Dtests.method="testCanRenewRetentionLeaseUnderBlock" \
  -Dtests.security.manager=true \
  -Dtests.locale=pa-Guru-IN \
  -Dtests.timezone=Pacific/Tahiti \
  -Dcompiler.java=12 \
  -Druntime.java=11

java.lang.AssertionError: ElasticsearchException[java.io.IOException: failed to read [id:1, file:/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/adoptopenjdk11/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_863577CB03E7FAE8-001/tempDir-008/data/nodes/1/indices/21IF9ApeTXuRqmqUckFHCw/0/_state/retention-leases-1.st]]; nested: IOException[failed to read [id:1, file:/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/adoptopenjdk11/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_863577CB03E7FAE8-001/tempDir-008/data/nodes/1/indices/21IF9ApeTXuRqmqUckFHCw/0/_state/retention-leases-1.st]]; nested: NoSuchFileException[/dev/shm/elastic+elasticsearch+6.8+matrix-java-periodic/ES_RUNTIME_JAVA/adoptopenjdk11/nodes/general-purpose/server/build/testrun/integTest/J3/temp/org.elasticsearch.index.seqno.RetentionLeaseIT_863577CB03E7FAE8-001/tempDir-008/data/nodes/1/indices/21IF9ApeTXuRqmqUckFHCw/0/_state/retention-leases-1.st];
	at __randomizedtesting.SeedInfo.seed([863577CB03E7FAE8:2EFEE0AAD0D3D279]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.lambda$testCanRenewRetentionLeaseUnderBlock$27(RetentionLeaseIT.java:482)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.runUnderBlockTest(RetentionLeaseIT.java:554)
	at org.elasticsearch.index.seqno.RetentionLeaseIT.testCanRenewRetentionLeaseUnderBlock(RetentionLeaseIT.java:464)

Neither failure reproduced locally for me.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[dnhatn]

@DaveCTurner Can you please take a look at these failures? Thank you.

[DaveCTurner]

This looks to be the same issue as #41430 -- the fix (#42620) was only backported to 7.2.0. I will backport it to 6.8.