MasterServiceTests.testThrottlingForMultipleTaskTypes

[dreamer-89]

Coming from #5955 (comment), this test seems to be failing for long with a decent frequency as captured here

> Task :server:test

org.opensearch.cluster.service.MasterServiceTests > testThrottlingForMultipleTaskTypes FAILED
    java.lang.AssertionError: expected:<9> but was:<6>
        at __randomizedtesting.SeedInfo.seed([FBB9A9A0370DCB37:89C5BECD42B90DAB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes(MasterServiceTests.java:958)


REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes" -Dtests.seed=FBB9A9A0370DCB37 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=be -Dtests.timezone=Pacific/Tarawa -Druntime.java=17

org.opensearch.cluster.service.MasterServiceTests > classMethod FAILED
    com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.opensearch.cluster.service.MasterServiceTests: 
       1) Thread[id=82, name=opensearch[test_node][clusterManagerService#updateTask][T#1], state=WAITING, group=TGRP-MasterServiceTests]
            at java.****@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
            at java.****@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
            at java.****@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
            at java.****@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
            at java.****@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
            at java.****@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
            at java.****@17.0.5/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.****@17.0.5/java.lang.Thread.run(Thread.java:833)
        at __randomizedtesting.SeedInfo.seed([FBB9A9A0370DCB37]:0)

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.service.MasterServiceTests" -Dtests.seed=FBB9A9A0370DCB37 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=17

org.opensearch.cluster.service.MasterServiceTests > classMethod FAILED
    com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
       1) Thread[id=82, name=opensearch[test_node][clusterManagerService#updateTask][T#1], state=WAITING, group=TGRP-MasterServiceTests]
            at java.****@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
            at java.****@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
            at java.****@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
            at java.****@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
            at java.****@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
            at java.****@17.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
            at java.****@17.0.5/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
            at java.****@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.****@17.0.5/java.lang.Thread.run(Thread.java:833)
        at __randomizedtesting.SeedInfo.seed([FBB9A9A0370DCB37]:0)

[shwetathareja]

@dhwanilpatel can you please look into the test failure?