J9 failed to execute all JUnit tests in a test file (caused by initializationError)

[fuzzy000]

Java -version output

The issue happened in following versions:

openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
Eclipse OpenJ9 VM (build openj9-0.17.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191017_442 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 97b5ec8f383 based on jdk8u232-b09)

openjdk version "9.0.4-adoptopenjdk"
OpenJDK Runtime Environment (build 9.0.4-adoptopenjdk+12)
Eclipse OpenJ9 VM (build openj9-0.9.0, JRE 9 Linux amd64-64-Bit Compressed References 20180814_248 (JIT enabled, AOT enabled)
OpenJ9   - 24e53631
OMR      - fad6bf6e
JCL      - feec4d2ae based on jdk-9.0.4+12)

openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.17.0, JRE 11 Linux amd64-64-Bit Compressed References 20191016_358 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 2a7af5674b based on jdk-11.0.5+10)

Summary of problem

We made some changes to a class file in the Apache Ant project (DirectoryScanner.class), and let several JVMs run the corresponding JUnit test file (DirectoryScannerTest.class) to test that class.

Among 29 tests in that test file, the OpenJDK series find 27 fails and report the reason for each failed test. The simplified message is like:

1) testSetFollowLinks(org.apache.tools.ant.DirectoryScannerTest): AssertionError
2) testAlternateIncludeExclude(org.apache.tools.ant.DirectoryScannerTest): VerifyError
3) testAbsolute1(org.apache.tools.ant.DirectoryScannerTest): VerifyError
...
27) testPatternsDifferInCaseScanningInsensitive(org.apache.tools.ant.DirectoryScannerTest): VerifyError

J9 series just report 1 fail (report message is also simplified):

1) initializationError(org.apache.tools.ant.DirectoryScannerTest): VerifyError
java.lang.VerifyError: JVMCFRE072 entries not sorted in lookupswitch bytecode; class=org/apache/tools/ant/DirectoryScanner, method=setIncludes([Ljava/lang/String;)V, pc=78
Exception Details:
  Location:
    org/apache/tools/ant/DirectoryScanner.setIncludes([Ljava/lang/String;)V @78: JBlookupswitch
  Reason:
    Error exists in the bytecode.

The initializationError is thrown by JUnit before it actually executes the tests. We think the reason for this discrepancy is the same as this issue: #9323. Maybe J9 did some kind of verification during the loading phase and cause the JUnit to fail when initializing.

However, from the perspective of a user, when using JUnit tests, I prefer to know the detailed information about which tests are passed and what is the reason for each failed test. If JUnit failed to initialize, in this case, I won't even know there are 2 tests passed.

JUnit Test	OpenJDK	J9
testAbsolute5	pass	N/A
testIsExcludedDirectoryScanned	pass	N/A
testSetFollowLinks	AssertionError	N/A
testAlternateIncludeExclude	VerifyError	N/A
testAbsolute1	VerifyError	N/A
testAbsolute2	VerifyError	N/A
testAbsolute3	VerifyError	N/A
testAbsolute4	VerifyError	N/A
testRecursiveExcludes	VerifyError	N/A
testChildrenOfExcludedDirectory	VerifyError	N/A
testContentsExcluded	VerifyError	N/A
test1	VerifyError	N/A
test2	VerifyError	N/A
test3	VerifyError	N/A
testExcludeOneFile	VerifyError	N/A
test2ButCaseInsensitive	VerifyError	N/A
testOrderOfIncludePatternsIrrelevant	VerifyError	N/A
testAlternateExcludeInclude	VerifyError	N/A
testFullPathMatchesCaseSensitive	VerifyError	N/A
testAllowSymlinks	VerifyError	N/A
testExcludeHasPrecedence	VerifyError	N/A
testPatternsDifferInCaseScanningSensitive	VerifyError	N/A
testParentDiffersInCaseScanningSensitive	VerifyError	N/A
testProhibitSymlinks	VerifyError	N/A
testFullpathDiffersInCaseScanningInsensitive	VerifyError	N/A
testFullpathDiffersInCaseScanningSensitive	VerifyError	N/A
testParentDiffersInCaseScanningInsensitive	VerifyError	N/A
testFullPathMatchesCaseInsensitive	VerifyError	N/A
testPatternsDifferInCaseScanningInsensitive	VerifyError	N/A

Additionally, in JVM specification 8, it seems verification is done in the linking stage (§5.4). It does not mention anything about verification in creation and loading (§5.3).

Diagnostic files

Verify5.zip

1. Extract it
2. In directory Verify5, run command java -cp sootOutput/junit-ant/:hamcrest-core-1.3.jar:junit-4.12.jar org.junit.runner.JUnitCore org.apache.tools.ant.DirectoryScannerTest

Execution environment

• OS and version: Ubuntu 16.04.6 LTS
• CPU model: Intel(R) Xeon(R) CPU E5-4610 v4 @ 1.80GHz
• Number of CPU cores: 4CPUs, each has 10 cores
• Size of physical memory: 16384 MB * 20