[BUG] Flaky Integ Test

[dhrubo-os]

What is the bug?

testMLInferenceProcessorLocalModel fails sometimes in workflow, showing flaky behavior.

Details log from workflow:

org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT > testMLInferenceProcessorLocalModel STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorLocalModel" -Dtests.seed=473B6E2CBF83BE4B -Dtests.security.manager=false -Dtests.locale=es -Dtests.timezone=Brazil/West -Druntime.java=11

REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorLocalModel" -Dtests.seed=473B6E2CBF83BE4B -Dtests.security.manager=false -Dtests.locale=es -Dtests.timezone=Brazil/West -Druntime.java=11

Suite: Test class org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorLocalModel" -Dtests.seed=473B6E2CBF83BE4B -Dtests.security.manager=false -Dtests.locale=es -Dtests.timezone=Brazil/West -Druntime.java=11
  2> org.junit.ComparisonFailure: expected:<DEPLOY[ED]> but was:<DEPLOY[ING]>
        at __randomizedtesting.SeedInfo.seed([473B6E2CBF83BE4B:35EABD6ECCC68677]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.lambda$testMLInferenceProcessorLocalModel$0(RestMLInferenceSearchRequestProcessorIT.java:278)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.verifyResponse(MLCommonsRestTestCase.java:671)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.getModel(MLCommonsRestTestCase.java:630)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.lambda$testMLInferenceProcessorLocalModel$1(RestMLInferenceSearchRequestProcessorIT.java:278)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.verifyResponse(MLCommonsRestTestCase.java:671)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.getTask(MLCommonsRestTestCase.java:635)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorLocalModel(RestMLInferenceSearchRequestProcessorIT.java:271)
  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT_473B6E2CBF83BE4B-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1914, maxMBSortInHeap=7.826473699175594, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es, timezone=Brazil/West
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 11.0.25 (64-bit)/cpus=4,threads=1,free=395628032,total=536870912
  2> NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestConnectorToolIT, RestMLCustomModelActionIT, RestMLCustomModelChunkActionIT, RestMLDeleteModelActionIT, RestMLDeleteModelGroupActionIT, RestMLDeleteTaskActionIT, RestMLDeployModelActionIT, RestMLFlowAgentIT, RestMLGetModelActionIT, RestMLGetModelGroupActionIT, RestMLGetTaskActionIT, RestMLGuardrailsIT, RestMLInferenceIngestProcessorIT, RestMLInferenceSearchRequestProcessorIT]


org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT > testMLInferenceProcessorLocalModel FAILED
    org.junit.ComparisonFailure: expected:<DEPLOY[ED]> but was:<DEPLOY[ING]>
        at __randomizedtesting.SeedInfo.seed([473B6E2CBF83BE4B:35EABD6ECCC68677]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.lambda$testMLInferenceProcessorLocalModel$0(RestMLInferenceSearchRequestProcessorIT.java:278)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.verifyResponse(MLCommonsRestTestCase.java:671)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.getModel(MLCommonsRestTestCase.java:630)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.lambda$testMLInferenceProcessorLocalModel$1(RestMLInferenceSearchRequestProcessorIT.java:278)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.verifyResponse(MLCommonsRestTestCase.java:671)
        at org.opensearch.ml.rest.MLCommonsRestTestCase.getTask(MLCommonsRestTestCase.java:635)
        at org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorLocalModel(RestMLInferenceSearchRequestProcessorIT.java:271)

org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT STANDARD_ERROR
    NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT_473B6E2CBF83BE4B-001

    NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1914, maxMBSortInHeap=7.826473699175594, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es, timezone=Brazil/West

    NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 11.0.25 (64-bit)/cpus=4,threads=1,free=395628032,total=536870912

    NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestConnectorToolIT, RestMLCustomModelActionIT, RestMLCustomModelChunkActionIT, RestMLDeleteModelActionIT, RestMLDeleteModelGroupActionIT, RestMLDeleteTaskActionIT, RestMLDeployModelActionIT, RestMLFlowAgentIT, RestMLGetModelActionIT, RestMLGetModelGroupActionIT, RestMLGetTaskActionIT, RestMLGuardrailsIT, RestMLInferenceIngestProcessorIT, RestMLInferenceSearchRequestProcessorIT]

We need to handle this situation

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.