KNNESSettingsTestIT failure in action "Build and Test k-NN Plugin"

[jmazanec15]

KNNESSettingsTestIT fails on Ubuntu during Github action. Additionally, was able to repro on Ubuntu 16:

com.amazon.opendistroforelasticsearch.knn.index.KNNESSettingsTestIT > testCacheRebuiltAfterUpdateIndexSettings FAILED
    java.lang.AssertionError: expected:<1> but was:<4>
        at __randomizedtesting.SeedInfo.seed([C11B19DAD31088ED:8E515F7F7E6EEC35]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.junit.Assert.assertEquals(Assert.java:631)
        at com.amazon.opendistroforelasticsearch.knn.index.KNNESSettingsTestIT.testCacheRebuiltAfterUpdateIndexSettings(KNNESSettingsTestIT.java:130)

The code that was being checked in was #170 , however, this does not impact any testing logic or that action logic. So, this is most likely a flaky test. I think it may be related to the tests in #162, even though the action for that PR did not fail.

[jmazanec15]

So, I think the issue is that the indices from another test are deleted, but not cleared from the cache, before the next test begins.

Indices for rest test cases are deleted after a test here

[jmazanec15]

For this, the cache should be cleared before the next test case starts. We can implement this by adding on to cleanUpCluster. When we have the clear cache API, we should use that to accomplish this. However, until then, we will need to use a workaround. One way to do this is to bounce one k-NN setting. This will result in an empty cache. @vamshin does this sound good?

[vamshin]

@jmazanec15 LGTM!
Lets leave a TODO comment on the test case and tag the issue to track the fix with clear cache api.

[jmazanec15]

Added comment here: #68 (comment)