GatewayMetaStatePersistedStateTests#testDataOnlyNodePersistence fails occasionally

[DaveCTurner]

Build scan:

https://gradle-enterprise.elastic.co/s/eh6lnrbtyc2y6

Repro line:

./gradlew ':server:test' --tests "org.elasticsearch.gateway.GatewayMetaStatePersistedStateTests.testDataOnlyNodePersistence" -Dtests.seed=6D2A353CEE651915 -Dtests.security.manager=true -Dtests.locale=sq -Dtests.timezone=Pacific/Apia -Dcompiler.java=14 -Druntime.java=8

Reproduces locally?:

Not tried, I'll work on this issue later.

Applicable branches:

I see similar failures on 7.6, 7.7, 7.8, 7.x and master.

Failure excerpt:

org.elasticsearch.gateway.GatewayMetaStatePersistedStateTests > testDataOnlyNodePersistence FAILED |  
-- | --
  | java.lang.AssertionError: expected:<2028814238962705350> but was:<0> |  
  | at __randomizedtesting.SeedInfo.seed([6D2A353CEE651915:D804B3D97E124E07]:0) |  
  | at org.junit.Assert.fail(Assert.java:88) |  
  | at org.junit.Assert.failNotEquals(Assert.java:834) |  
  | at org.junit.Assert.assertEquals(Assert.java:645) |  
  | at org.junit.Assert.assertEquals(Assert.java:631) |  
  | at org.elasticsearch.gateway.GatewayMetaStatePersistedStateTests.testDataOnlyNodePersistence(GatewayMetaStatePersistedStateTests.java:406) |  
  | REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.gateway.GatewayMetaStatePersistedStateTests.testDataOnlyNodePersistence" -Dtests.seed=6D2A353CEE651915 -Dtests.security.manager=true -Dtests.locale=sq -Dtests.timezone=Pacific/Apia -Dcompiler.java=14 -Druntime.java=8

Looks to duplicate #53193 which was attributed to #53195, but as it's still ongoing I am opening this issue again to investigate further.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)