Allow VectorUtilProvider tests to be executed although hardware may not fully support vectorization or if C2 is not enabled

[uschindler]

This PR allows us to better test the new Panama Vector Support, although the hardware may not fully support it. Currently when developers are running tests, C2 is disabled, so the provider never enables itsself. Also CPUs only with AVX1 are disabled.

To actually test the correctness of the provider with unsupported hardware, at the moment developers have to pass export CI=true or similar to enable C2 which slows down tests.

This PR adds a boolean to lookupProvider() called testMode which enforces its usage for tests (which directly test provider interface). The code is only used during tests, actually only in TestVectorUtilProviders, e.g. see here (the second test prints "test mode" while loading):

$ gradlew :lucene:core:test --tests 'TestVectorUtil*' -Ptests.verbose=true
> Task :lucene:core:test
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release
  2> Juni 15, 2023 5:58:12 PM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
  2> INFORMATION: Using MemorySegmentIndexInput with Java 20; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
  1> filesystem: ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(VerboseFS(sun.nio.fs.WindowsFileSystemProvider@76caf9d5))))))
  1> FS 0 [2023-06-15T15:58:13.110199200Z; SUITE-TestVectorUtil-seed#[B8B8B3EB0F1123E8]-worker]: createDirectory: ..\tests-tmp (FAILED: java.nio.file.FileAlreadyExistsException: C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\core\build\tmp\tests-tmp)
  1> Loaded codecs: [MinimalCodec, MinimalCompoundCodec, Lucene95, Asserting, CheapBastard, DeflateWithPresetCompressingStoredFieldsData, FastCompressingStoredFieldsData, FastDecompressionCompressingStoredFieldsData, HighCompressionCompressingStoredFieldsData, LZ4WithPresetCompressingStoredFieldsData, DummyCompressingStoredFieldsData, ConfigurableMCodec, SimpleText]
  1> Loaded postingsFormats: [Lucene90, MockRandom, RAMOnly, LuceneFixedGap, LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval, TestBloomFilteredLucenePostings, Asserting, UniformSplitRot13, STUniformSplitRot13, BlockTreeOrds, BloomFilter, Direct, FST50, UniformSplit, SharedTermsUniformSplit]
  2> śuw 15, 2023 3:58:13 ALUULA org.apache.lucene.util.VectorUtilProvider lookup
  2> WARNING: C2 compiler is disabled; Java vector incubator API can't be enabled
  2> NOTE: test params are: codec=Asserting(Lucene95), sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=khq-Latn-ML, timezone=Etc/GMT0
  2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 20 (64-bit)/cpus=1,threads=1,free=220378024,total=257425408
  2> NOTE: All tests run in this JVM: [TestVectorUtil]
  2> Juni 15, 2023 5:58:14 PM org.apache.lucene.util.VectorUtilPanamaProvider <init>
  2> INFORMATION: Java vector incubator API enabled (test mode); uses preferredBitSize=256
  1> filesystem: ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(VerboseFS(sun.nio.fs.WindowsFileSystemProvider@76caf9d5))))))
  1> FS 1 [2023-06-15T15:58:14.169918700Z; SUITE-TestVectorUtilProviders-seed#[B8B8B3EB0F1123E8]-worker]: createDirectory: ..\tests-tmp (FAILED: java.nio.file.FileAlreadyExistsException: C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\core\build\tmp\tests-tmp)
  1> Loaded codecs: [MinimalCodec, MinimalCompoundCodec, Lucene95, Asserting, CheapBastard, DeflateWithPresetCompressingStoredFieldsData, FastCompressingStoredFieldsData, FastDecompressionCompressingStoredFieldsData, HighCompressionCompressingStoredFieldsData, LZ4WithPresetCompressingStoredFieldsData, DummyCompressingStoredFieldsData, ConfigurableMCodec, SimpleText]
  1> Loaded postingsFormats: [Lucene90, MockRandom, RAMOnly, LuceneFixedGap, LuceneVarGapFixedInterval, LuceneVarGapDocFreqInterval, TestBloomFilteredLucenePostings, Asserting, UniformSplitRot13, STUniformSplitRot13, BlockTreeOrds, BloomFilter, Direct, FST50, UniformSplit, SharedTermsUniformSplit]
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=2013, maxMBSortInHeap=5.179808517789641, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=nyn-UG, timezone=America/Indiana/Petersburg
  2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 20 (64-bit)/cpus=1,threads=1,free=277798920,total=324534272
  2> NOTE: All tests run in this JVM: [TestVectorUtil, TestVectorUtilProviders]
:lucene:core:test (SUCCESS): 57 test(s)
The slowest suites (exceeding 1s) during this run:
   1.85s TestVectorUtil (:lucene:core)
   1.03s TestVectorUtilProviders (:lucene:core)

[ChrisHegarty]

It is a bit of a pity that this check is now with a final rather than static final, but that's the price we pay for test-ability!

[uschindler]

Thats ok, because in production we only have exactly one instance of this class, so it does not matter, it is optimized away.

[uschindler]

Actually I like this more, because it is no negative test.

[ChrisHegarty]

This is great - that we can ensure that we're always testing the Panama implementation. I guess the test can sometime be slow?

[uschindler]

This test only compares the default lucene provider with a few random vectors and compares results. So slowness does not matter here!

Of course here we only test the Panama Implementation if it is Java 20/21 and it is not buggy (turkish locale bug).

[uschindler]

Actually I would like to have this testability enabled, because it is planned to run Smoke Tester for release also with Java 19/20/21 if the tester enables it, but as the tests run in normal "developer mode", the whole thing won't be tested.

I need to open issue for this. At moment I don't like it that this code is not tested by the actual release manager... :-(

Maybe for speed improvement we can make Smoke tester to only run a few tests (like above) with 19, 20, 21.

[rmuir]

i guess my main question would be, what are we testing?

With this approach, we just test the jdk.incubator.vector slow fallback code. Wouldn't it be better to just enable C2/export CI in jenkins? We could e.g. consider randomizing UseAVX to better exercise it there, too. I think this would provide more value than thoroughly testing the slow fallback code?

[rmuir]

especially since we work pretty hard to prevent the slow fallback code from ever executing for real :)

[uschindler]

I's just a unit test! We just test that our code is semantically correct. Of course we assume that the fallback code is correct. So if we for example add some wrong sign flip it will fail.

Theres nothing else we test. If Hotspot fails its not our turn to test it.

[ChrisHegarty]

This is just a very basic sanity of the code, which I think is fine. I’d find it useful to have this if modifying the code locally. The CI, etc should already be testing this more extensively.

Separately, how much time is disabling C2 saving when running unit tests?

[uschindler]

This is just a very basic sanity of the code, which I think is fine. I’d find it useful to have this if modifying the code locally. The CI, etc should already be testing this more extensively.

Thanks. The main idea of this test is:

• make sure the code actuall links and works with JDK 20/21 API (as we only compile against stubs).
• make sure the code behaves identical to the scalar impl (I trust Roberts code, but what happens if somebody changes a variable name in the code). In addition this improves test coverage... We do not want to test Panama Vector nor Hotspot here!

Separately, how much time is disabling C2 saving when running unit tests?

We had several discussions, but it was dramatical for tests. For the short living execs during build (like javac, javadoc, ecj) we have: #10900; for tests we had a separate issue, not sure which one.

[nknize]

Curious what I'm missing here. With the jdk set to 20 checkVersion in WrapperDownloader.java is gating the execution?

CI=true ./gradlew :lucene:core:test --tests "org.apache.lucene.internal.vectorization.TestVectorUtilSupport" --info
ERROR: java version must be between 17 and 19, your version: 20

Is there a local change to bypass this? Just upping the version gives:

* What went wrong:
java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /home/nknize/.gradle/caches/jars-9/0b5cc9218fa11343299cd00875d16caf/buildSrc.jar.
> org.gradle.api.GradleException: Failed to create Jar file....

Seems running this simple test requires some gymnastics here? Or am I missing something?

[uschindler]

You can't run Gradle with that version as it's incompatible. This has nothing to do with this PR, it is documented in Lucene's help: https://github.com/apache/lucene/blob/main/help/jvms.txt

In short: To run tests with some specific version use:

• env var RUNTIME_JAVA_HOME
• or pass a prop: -Pruntime.java.home=...

This allows us to run Jenkins with any version our there (als Java EA releases which aren't supported by Gradle at all).

[nknize]

• pass a prop: -Pruntime.java.home=...

That's what I missed. I was using -Druntime.java=20 which does nothing. :) Thanks @uschindler!