Flaky tests for KNNCodecServiceTests

[jmazanec15]

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.codec.KNNCodecServiceTests.testGetCodecByName" -Dtests.seed=7D83598ACDB3A41 -Dtests.security.manager=false -Dtests.locale=es-ES -Dtests.timezone=Europe/Andorra -Druntime.java=21


Suite: Test class org.opensearch.knn.index.codec.KNNCodecServiceTests
  2> REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.codec.KNNCodecServiceTests.testGetCodecByNameWithNoMapperService" -Dtests.seed=7D83598ACDB3A41 -Dtests.security.manager=false -Dtests.locale=es-ES -Dtests.timezone=Europe/Andorra -Druntime.java=21
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([7D83598ACDB3A41:2E386DBC807A0A19]:0)
        at org.opensearch.index.IndexSettings.determineRemoteStorePathStrategy(IndexSettings.java:1916)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:993)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:898)
        at org.opensearch.knn.index.codec.KNNCodecServiceTests.setUp(KNNCodecServiceTests.java:41)
  2> REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.codec.KNNCodecServiceTests.testGetCodecByName" -Dtests.seed=7D83598ACDB3A41 -Dtests.security.manager=false -Dtests.locale=es-ES -Dtests.timezone=Europe/Andorra -Druntime.java=21
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([7D83598ACDB3A41:695FD9C6CB0F6A55]:0)
        at org.opensearch.index.IndexSettings.determineRemoteStorePathStrategy(IndexSettings.java:1916)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:993)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:898)
        at org.opensearch.knn.index.codec.KNNCodecServiceTests.setUp(KNNCodecServiceTests.java:41)
  2> NOTE: leaving temporary files on disk at: /__w/k-NN/k-NN/build/testrun/test/temp/org.opensearch.knn.index.codec.KNNCodecServiceTests_7D83598ACDB3A41-001
  2> NOTE: test params are: codec=Asserting(Lucene99): {}, docValues:{}, maxPointsInLeafNode=1481, maxMBSortInHeap=7.284359240009941, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=es-ES, timezone=Europe/Andorra
  2> NOTE: Linux 6.5.0-1017-azure amd64/Azul Systems, Inc. 21.0.2 (64-bit)/cpus=4,threads=1,free=491220200,total=536870912
  2> NOTE: All tests run in this JVM: [KNNInvalidIndicesExceptionTests, IndexUtilTests, KNNCreateIndexFromModelTests, KNNMethodContextTests, KNNSettingsTests, KNNVectorIndexFieldDataTests, MethodComponentContextTests, ParameterTests, VectorDataTypeTests, KNNCodecServiceTests]
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([7D83598ACDB3A41:2E386DBC807A0A19]:0)
        at org.opensearch.index.IndexSettings.determineRemoteStorePathStrategy(IndexSettings.java:1916)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:993)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:898)
        at org.opensearch.knn.index.codec.KNNCodecServiceTests.setUp(KNNCodecServiceTests.java:41)

org.opensearch.knn.index.codec.KNNCodecServiceTests > testGetCodecByName FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([7D83598ACDB3A41:695FD9C6CB0F6A55]:0)
        at org.opensearch.index.IndexSettings.determineRemoteStorePathStrategy(IndexSettings.java:1916)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:993)
        at org.opensearch.index.IndexSettings.<init>(IndexSettings.java:898)
        at org.opensearch.knn.index.codec.KNNCodecServiceTests.setUp(KNNCodecServiceTests.java:41)
  1> [2024-04-05T18:11:00,537][INFO ][o.o.k.i.c.KNNCodecServiceTests] [testGetCodecByNameWithNoMapperService] after test
  1> [2024-04-05T18:11:00,637][INFO ][o.o.k.i.c.KNNCodecServiceTests] [testGetCodecByName] after test

WARNING clustering 100 points to 5 centroids: please provide at least 195 training points

> Task :test

Tests with failures:
 - org.opensearch.knn.index.codec.KNNCodecServiceTests.testGetCodecByNameWithNoMapperService
 - org.opensearch.knn.index.codec.KNNCodecServiceTests.testGetCodecByName

527 tests completed, 2 failed, 2 skipped

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///__w/k-NN/k-NN/build/reports/tests/test/index.html

* Try:
> Run with --scan to get full insights.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.


BUILD FAILED in 20m 3s
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
37 actionable tasks: 37 executed
Error: Process completed with exit code 1.

Related:

1. https://github.com/opensearch-project/k-NN/actions/runs/8572401820/job/23494871195?pr=1592

[navneet1v]

All we need to do to fix these issue is add this line:

when(indexMetadata.getCustomData(IndexMetadata.REMOTE_STORE_CUSTOM_KEY)).thenReturn(null);

here:

k-NN/src/test/java/org/opensearch/knn/index/codec/KNNCodecServiceTests.java

Lines 37 to 40 in 98b6218

	IndexMetadata indexMetadata = mock(IndexMetadata.class);
	when(indexMetadata.getIndex()).thenReturn(new Index(TEST_INDEX, INDEX_UUID.toString()));
	when(indexMetadata.getSettings()).thenReturn(Settings.EMPTY);
	Settings settings = Settings.builder().put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, Integer.toString(NUM_OF_SHARDS)).build();

[ryanbogan]

Seeing the flaky tests on 2.x for this PR: https://github.com/opensearch-project/k-NN/actions/runs/8884598082/attempts/1?pr=1661

@navneet1v I see that the above fix was never backported to 2.x. Is that intentional?

[navneet1v]

@ryanbogan at that time the issue was only with main branch hence the PR #1601 was not backported. We can backport the PR now too.

[navneet1v]

@ryanbogan I created the backport: #1663

[ryanbogan]

Backport merged and tests are passing