Functional tests do not compile on z/OS #660

Mesbah-Alam · 2018-10-24T19:10:45Z

When functional teat compilation is attempted on z/OS, we get the following type of errors:

11:35:49     [javac] /home/jenkins/workspace/Test-sanity.functional-JDK11-zos_390-64_cmprssptrs/openjdk-tests/functional/IllegalAccessError_for_protected_method/src/org/openj9/test/Agent.java:1: error: illegal character: '\u0015'

I suspect this is due to the fact that we are not supplying the correct encoding value in the build.xml files of the functional tests.

We need to define what file encoding to be used for Java source files for tests as we do in case of system tests.

Build link: https://hyc-runtimes-jenkins.swg-devops.com/view/Test_functional/job/Test-sanity.functional-JDK11-zos_390-64_cmprssptrs/9/console

The text was updated successfully, but these errors were encountered:

Mesbah-Alam · 2018-10-24T19:12:15Z

FYI @sophia-guo @vsebe

Mesbah-Alam · 2018-10-24T19:22:29Z

Functional tests are using encoding="ISO-8859-1" (e.g. jsr292).

We need to update this and use IBM-1047 while on z/OS.

Mesbah-Alam · 2018-11-28T03:30:06Z

FYI @llxia - this is the issue i was speaking about earlier.

Mesbah-Alam · 2018-11-28T20:19:58Z

For system tests we have a parameter defined in top.xml that sets source encoding to IBM-1047 when on z/OS, and tp UTF-8 otherwise.

<condition property="src-encoding" value="IBM-1047" else="UTF-8">
		<isset property="isZOS"/>
</condition>

https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/2fb744f0c847bb31f6b4aa39173a3977f523008c/openjdk.build/include/top.xml#L381

The src-encoding property is subsequently used in the test projects' build files in the javac task:

<javac srcdir="${src_dir}"
			   destdir="${bin_dir}"
			   fork="true"
			   executable="${java8_compiler}"
			   debug="true"
			   classpathref="project.class.path"
			   encoding="${src-encoding}"
			   includeantruntime="false"
			   failonerror="true">
			<include name="**/*.java"/>
</javac>

https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/2fb744f0c847bb31f6b4aa39173a3977f523008c/openjdk.test.jlm/build.xml#L82

We need to have something similar for functional tests too, for them to compile and run on z/OS. May be we could have TestConfig set a global parameter that could be used by all tests?

@llxia @renfeiw

llxia · 2018-11-28T20:51:17Z

I am curious to know why we set encoding in the first place. It is not required attribute. What is the default encoding value used on zos?

Mesbah-Alam · 2018-11-28T21:08:20Z

IBM-1047 should be the default value for any text files on zos. It is likely used by Java compiler to determine file encoding of the Java source it compiles. You can try to run the build without any encoding value set to see if it works both on z/os or non-z/os environment.

llxia · 2018-11-29T13:46:16Z

Without changing anything, below is what I observed:

It seems test material is not converted in ebcdic on zos. @Mesbah-Alam Do we convert files into ebcdic?
TKG has issue to parse ottawa.csv (I think this is related to files are not in ebcdic)

00:01:00.172 perl configure.pl
00:01:00.172 cd /home/jenkins/workspace/Grinder/openjdk-tests/TestConfig/scripts/testKitGen; \
00:01:00.172 perl testKitGen.pl --graphSpecs=zos_390-64_cmprssptrs --jdkVersion=8 --impl=openj9 --buildList=functional/JIT_Test ; \
00:01:00.172 cd /home/jenkins/workspace/Grinder/openjdk-tests/TestConfig;
00:01:00.172 Line could not be parsed: Variables,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
00:01:00.172 
00:01:00.172 Line could not be parsed: ,win_x86,win_x86_purec,win_x86-64,win_x86-64_purec,win_x86-64_cmprssptrs,win_x86-

Not all functaional tests failed during compilation. I tried with a single test finalizerTest in functional/JIT_Test and it passed.

00:02:55.291 Running make 4.1
00:02:55.291 set TEST_ROOT to /home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../..
00:02:55.291 set JDK_VERSION to 8
00:02:55.291 set JDK_IMPL to openj9
00:02:55.291 set JVM_VERSION to openjdk8-openj9
00:02:55.291 set JCL_VERSION to latest
00:02:55.291 set JAVA_BIN to /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin
00:02:55.291 set SPEC to zos_390-64_cmprssptrs
00:02:55.291 Running finalizerTest ...
00:02:55.291 "/home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/java" -version
00:02:55.291 JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
00:02:55.928 java version "1.8.0_201"
00:02:55.928 Java(TM) SE Runtime Environment (build 8.0.6.0 - pmz6480sr6-20181127_01(SR6))
00:02:55.928 IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20181126_403391 (JIT enabled, AOT enabled)
00:02:55.928 OpenJ9   - 4bb869d
00:02:55.928 OMR      - 7d269b1
00:02:55.928 IBM      - e2996d1)
00:02:55.928 JCL - 20181121_01 based on Oracle jdk8u201-b05
00:02:55.928 make[1]: Entering directory '/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/functional'
00:02:55.928 make[2]: Entering directory '/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/functional/JIT_Test'
00:02:55.928 
00:02:55.928 ===============================================
00:02:55.928 Running test finalizerTest_0 ...
00:02:55.928 ===============================================
00:02:55.928 finalizerTest_0 Start Time: Wed Nov 28 21:45:39 2018 Epoch Time (ms): 1543441539159
00:02:55.928 variation: Mode100
00:02:55.928 JVM_OPTIONS: -Xcompressedrefs -Xint -Xgcpolicy:optthruput -Xnocompressedrefs 
00:02:55.928 { mkdir -p "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15434415385650/finalizerTest_0"; \
00:02:55.928 cd "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15434415385650/finalizerTest_0"; \
00:02:55.928 "/home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/java" -Xcompressedrefs -Xint -Xgcpolicy:optthruput -Xnocompressedrefs  \
00:02:55.929 -cp "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/resources:/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/lib/testng.jar:/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/lib/jcommander.jar:/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/JIT_Test/jitt.jar" \
00:02:55.929 org.testng.TestNG -d "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15434415385650/finalizerTest_0" "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/JIT_Test/testng.xml" \
00:02:55.929 -testnames finalizerTest -groups level.sanity \
00:02:55.929 -excludegroups d.*.zos_390-64_cmprssptrs,d.*.arch.390,d.*.os.zos,d.*.bits.64,d.*.generic-all; \
00:02:55.929 if [ $? -eq 0 ] ; then echo ""; echo "finalizerTest_0""_PASSED"; echo ""; cd /home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../..;  else echo ""; echo "finalizerTest_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15434415385650/TestTargetResult";
00:02:55.929 JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
00:02:57.978 [IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/resources/excludes/latest_exclude_8.txt
00:02:57.978 ...
00:02:57.978 ... TestNG 6.14.2 by Cédric Beust ([email protected])
00:02:57.978 ...
00:02:57.978 
00:02:57.978 [TestObject] [INFO] TestObject working!
00:02:57.978 [TestObject] [INFO] Done!
00:03:00.047 [TestObject] [INFO] PASSED:FINALIZER_INVOKED
00:03:00.047 PASSED: testFinalizer
00:03:00.047 
00:03:00.047 ===============================================
00:03:00.047     finalizerTest
00:03:00.047     Tests run: 1, Failures: 0, Skips: 0
00:03:00.047 ===============================================
00:03:00.047 
00:03:00.047 
00:03:00.047 ===============================================
00:03:00.047 JIT_Test suite
00:03:00.047 Total tests run: 1, Failures: 0, Skips: 0
00:03:00.047 ===============================================
00:03:00.047 
00:03:03.424 
00:03:03.424 finalizerTest_0_PASSED
00:03:03.424 
00:03:03.424 finalizerTest_0 Finish Time: Wed Nov 28 21:45:44 2018 Epoch Time (ms): 1543441544907
00:03:03.424 make[2]: Leaving directory '/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/functional/JIT_Test'
00:03:03.424 make[1]: Leaving directory '/home/jenkins/workspace/Grinder_Sandbox/openjdk-tests/functional'
00:03:03.424 
00:03:03.424 
00:03:03.424 TEST TARGETS SUMMARY
00:03:03.424 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
00:03:03.424 PASSED test targets:
00:03:03.424 	finalizerTest_0
00:03:03.424 
00:03:03.424 TOTAL: 1   EXECUTED: 1   PASSED: 1   FAILED: 0   SKIPPED: 0
00:03:03.424 ALL TESTS PASSED
00:03:03.424 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TAP plugin seems has issue to read the file on master. File can be found on slave machine /home/jenkins/workspace/Grinder/openjdk-tests/TestConfig/test_output_15434974007394 and it looks fine.

00:01:25.506 Looking for TAP results report in workspace using pattern: **/*.tap
00:01:25.788 Saving reports...
00:01:26.103 Processing '/var/jenkins_home/jobs/Grinder/builds/874/tap-master-files/openjdk-tests/TestConfig/test_output_15434974007394/TestTargetResult.tap'
00:01:26.104 Parsing TAP test result [/var/jenkins_home/jobs/Grinder/builds/874/tap-master-files/openjdk-tests/TestConfig/test_output_15434974007394/TestTargetResult.tap].
00:01:26.104 org.tap4j.parser.ParserException: Error parsing TAP Stream: Missing TAP Plan.

Mesbah-Alam · 2018-11-29T14:32:56Z

Ebcdic conversation is done by Git itself when it checks out code on z/OS. For this, each of the 3 systemtest repos has defined its own .gitattriutes file (e.g. https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/master/.gitattributes) where we define which file should be left alone and which file should be converted. This file is used in the git command, see: https://github.com/AdoptOpenJDK/openjdk-tests/blob/ac0012d53f1223456c4b8dfc9a5a99675aa279ba/systemtest/build.xml#L46 - this ensures that text files are converted, by git, into ebcdic.

In the case of functional tests, I believe we are getting lucky in the case where the tests are running.

llxia · 2018-11-29T15:02:22Z

Even in system test, not all files are converted to ebcdic:

ZOSCAN03:/home/jenkins/workspace/Grinder/openjdk-tests/systemtest/openjdk-systemtest/openjdk.test.concurrent>cat build.xml
�_%������?>���������<���>�����>��ʀ������/����<���>���������?>���������<���>������`?̀_/`�>?Ȁ�������ˀ��%�������Ȁ�>��?_�%�/>������ǀ���

This may explain in system test console output, we see following:

00:05:44.192 setup-java8-properties:
00:05:44.192      [echo] java.home is /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/..
00:05:44.192      [echo] java8_bindir set to /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/../../bin
00:05:44.192      [echo] java8_compiler set to /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/javac
00:05:44.192      [echo] Running /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -XshowSettings:properties -version to determine java platform
00:05:44.192 [echoproperties] � >È�øÊ?øÁÊÈÑÁË��ïÁÀ�+?Î�������������áëè������¦/Î/�����¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^������%ÑÂ�Ë���Ì��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�ÑÄÄ��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì�Ä?_øÊÁËËÁÀÊÁÃË��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì�¦�Î_��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�Ï?Ê,Ëø/
...

Mesbah-Alam · 2018-11-29T16:16:58Z

Yes, I have seen these special characters in the job output as well, I am not sure if it's because some files are not converted in ebcdic (because if not, we would see tests not compiling / running). For example, the following seems to be a z/OS Ant bug:

00:05:44.192      [echo] Running /home/jenkins/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -XshowSettings:properties -version to determine java platform
00:05:44.192 [echoproperties] � >È�øÊ?øÁÊÈÑÁË��ïÁÀ�+?Î�������������áëè������¦/Î/�����¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^������%ÑÂ�Ë���Ì��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�ÑÄÄ��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì�Ä?_øÊÁËËÁÀÊÁÃË��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�?øÁ>¦�^ÊÁË?ÍÊÄÁË�¢���^���%ÑÂ�Ë���Ì�¦�Î_��¦/Î/���Ç?_Á�¦Á>,Ñ>Ë�Ï?Ê,Ëø/

Note also that, text files do not need to be in 'ebcdic' format for the new z/OS to process them either. You can transfer ascii files on a z/OS machine, and open them. It's the JVM that requires texts files to be in ebcdic format while on z/OS.

vsebe · 2018-11-29T16:28:06Z

The files are correctly tagged after git clone. Looks the files are not tagged after the copy from git clone directory to jvmtest directory.

llxia · 2018-11-29T17:02:20Z

Thanks for @vsebe help!

@Mesbah-Alam, @renfeiw and I had the discussion. We actually have mutiple issues on zos. We will work on the following:

ottawa.csv cannot be parsed (@renfeiw )
some fucntional test does copy in ant which does not copy the proper tag. (@llxia )
TAP file cannot be processed in master (@renfeiw )
system test output contains special characters (@Mesbah-Alam )

llxia · 2018-12-04T12:58:36Z

Just for the record, encoding is not needed for javac if the files are properly tagged on zos. The files are correctly tagged after git clone. Therefore, encodingis not needed forjavac` in most cases.

However, in a couple of our tests, we do ant <copy> after git clone. And we noticed that the files are not tagged correctly after ant <copy>. In these cases, encoding is needed for javac in order to read the untagged files properly.

renfeiw · 2018-12-04T15:30:54Z

The Tap file issue should be fixed: #722

smlambert · 2019-01-17T18:28:30Z

Part of the problem is being fixed upstream, are there any other actions that need to be completed before we close this?

llxia · 2019-01-18T13:35:13Z

some fucntional test does copy in ant which does not copy the proper tag. (@llxia )
TAP file cannot be processed in master (@renfeiw )

Above two issues are fixed. Also as Shelley mentioned, compilation issue is being fixed upstream. I will close this issue. Once we get new SDK with the fix from upstream, we can re-test and open issue as needed.

Mesbah-Alam added the bug label Oct 24, 2018

This was referenced Nov 30, 2018

set javac encoding eclipse-openj9/openj9#3900

Merged

Convert tap file from ebcdic to ascii on zos #722

Merged

llxia closed this as completed Jan 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Functional tests do not compile on z/OS #660

Functional tests do not compile on z/OS #660

Mesbah-Alam commented Oct 24, 2018 •

edited

Loading

Mesbah-Alam commented Oct 24, 2018

Mesbah-Alam commented Oct 24, 2018 •

edited

Loading

Mesbah-Alam commented Nov 28, 2018

Mesbah-Alam commented Nov 28, 2018 •

edited

Loading

llxia commented Nov 28, 2018

Mesbah-Alam commented Nov 28, 2018 •

edited

Loading

llxia commented Nov 29, 2018 •

edited

Loading

Mesbah-Alam commented Nov 29, 2018

llxia commented Nov 29, 2018

Mesbah-Alam commented Nov 29, 2018 •

edited

Loading

vsebe commented Nov 29, 2018

llxia commented Nov 29, 2018

llxia commented Dec 4, 2018 •

edited

Loading

renfeiw commented Dec 4, 2018

smlambert commented Jan 17, 2019

llxia commented Jan 18, 2019

Functional tests do not compile on z/OS #660

Functional tests do not compile on z/OS #660

Comments

Mesbah-Alam commented Oct 24, 2018 • edited Loading

Mesbah-Alam commented Oct 24, 2018

Mesbah-Alam commented Oct 24, 2018 • edited Loading

Mesbah-Alam commented Nov 28, 2018

Mesbah-Alam commented Nov 28, 2018 • edited Loading

llxia commented Nov 28, 2018

Mesbah-Alam commented Nov 28, 2018 • edited Loading

llxia commented Nov 29, 2018 • edited Loading

Mesbah-Alam commented Nov 29, 2018

llxia commented Nov 29, 2018

Mesbah-Alam commented Nov 29, 2018 • edited Loading

vsebe commented Nov 29, 2018

llxia commented Nov 29, 2018

llxia commented Dec 4, 2018 • edited Loading

renfeiw commented Dec 4, 2018

smlambert commented Jan 17, 2019

llxia commented Jan 18, 2019

Mesbah-Alam commented Oct 24, 2018 •

edited

Loading

Mesbah-Alam commented Oct 24, 2018 •

edited

Loading

Mesbah-Alam commented Nov 28, 2018 •

edited

Loading

Mesbah-Alam commented Nov 28, 2018 •

edited

Loading

llxia commented Nov 29, 2018 •

edited

Loading

Mesbah-Alam commented Nov 29, 2018 •

edited

Loading

llxia commented Dec 4, 2018 •

edited

Loading