[CI] SnapshotBasedIndexRecoveryIT testRecoveryIsCancelledAfterDeletingTheIndex failing

[benwtrent]

This is a weird timeout issue. From what I can tell there are no related issues. Seems suspect that its on arm.

Build scan:
https://gradle-enterprise.elastic.co/s/exqmn7xel4bcg/tests/:server:internalClusterTest/org.elasticsearch.indices.recovery.SnapshotBasedIndexRecoveryIT/testRecoveryIsCancelledAfterDeletingTheIndex

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.recovery.SnapshotBasedIndexRecoveryIT.testRecoveryIsCancelledAfterDeletingTheIndex" -Dtests.seed=E36E1308685B0AF6 -Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH -Dtests.timezone=Indian/Reunion -Druntime.java=16

Applicable branches:
7.x

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.indices.recovery.SnapshotBasedIndexRecoveryIT&tests.test=testRecoveryIsCancelledAfterDeletingTheIndex

Failure excerpt:

java.lang.Exception: Test abandoned because suite timeout was reached.

  at __randomizedtesting.SeedInfo.seed([E36E1308685B0AF6]:0)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[benwtrent]

So, it may not be due to arm.

Had another failure: https://gradle-enterprise.elastic.co/s/y3rabg37c5mxm

Still linux, and still a timeout waiting for cluster green state for java.lang.AssertionError: timed out waiting for green state for index [snapshot_based_recovery] cluster state [{

[henningandersen]

Second failure reported above is for SnapshotBasedRecoveryIT.testSnapshotBasedRecovery, and looks identical to #76595

[davidkyle]

Another timeout, ARM again but this time the master branch - https://gradle-enterprise.elastic.co/s/hae4wi4wocnus