[CI] IndexFoldersDeletionListenerIT > testListenersInvokedWhenIndexIsDangling occassionally failing

[mark-vieira]

Looks like a newly added test that has failed a few times now.

Build scan:
https://gradle-enterprise.elastic.co/s/wits5fsx622y2/tests/:server:internalClusterTest/org.elasticsearch.plugins.IndexFoldersDeletionListenerIT/testListenersInvokedWhenIndexIsDangling#1

Repro line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.plugins.IndexFoldersDeletionListenerIT.testListenersInvokedWhenIndexIsDangling" -Dtests.seed=8A30512BA393F582 -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=EST -Druntime.java=11

Reproduces locally?:
Nope.

Applicable branches:
master and 7.x

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P28D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.plugins.IndexFoldersDeletionListenerIT&tests.disabledDistributions=WyJvdXRjb21lOnBhc3NlZCJd&tests.sortField=FAILED&tests.test=testListenersInvokedWhenIndexIsDangling&tests.unstableOnly=true

Failure excerpt:

org.elasticsearch.plugins.IndexFoldersDeletionListenerIT > testListenersInvokedWhenIndexIsDangling FAILED
    java.lang.AssertionError: Expecting no shards deleted on node [node_t3] but got: {[hokkjihfdt/ldTt4FW9Q6CLZ7hUXlrj4A]=[[hokkjihfdt][7]]}
        at __randomizedtesting.SeedInfo.seed([8A30512BA393F582:60BB557EF8D46F5D]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.elasticsearch.plugins.IndexFoldersDeletionListenerIT.assertNoDeletions(IndexFoldersDeletionListenerIT.java:368)
        at org.elasticsearch.plugins.IndexFoldersDeletionListenerIT.testListenersInvokedWhenIndexIsDangling(IndexFoldersDeletionListenerIT.java:214)

[mark-vieira]

@tlrx do you mind taking a look since you added this test recently? Looks like other tests in the same suite have failed a number of times as well.

[mark-vieira]

Looks like this is the same as #66519 but given this just failed again in master intake it seems the fix didn't completely eliminate the issue.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)