ibmjvmti.h not found in OpenJ9 SDK

[Mesbah-Alam]

Overview
There seems to be a bug in building OpenJ9 with the OpenJDK extensions. The code is included in the VM, but the ibmjvmti.h file is not included in the include directory.

Test failure

SharedClasses JVMTI test fails to compile with openjdk8-openj9 SDK

6:49:14 [exec] /home/jenkins/workspace/test_personal/openjdk-tests/systemtest/openj9-systemtest/openj9.test.sharedClasses.jvmti/src/native/sharedClasses.c:32:22: fatal error: ibmjvmti.h: No such file or directory

Link
https://ci.adoptopenjdk.net/view/work%20in%20progress/job/test_personal/280/console

How to reproduce
The error can be reproduced using a personal build in ci.Adopt, using the following steps:

1. Include the disabled failing test in openj9-systemtest:
   1.1) Create a fork and a branch of https://github.com/eclipse/openj9-systemtest/
   1.2) In your branch, reverse the following change in openj9.test.sharedClasses.jvmti/build.xml : https://github.com/eclipse/openj9-systemtest/pull/18/files (i.e., add the old version of the target called build, and remove the new version of the same where the if condition for "openj9" was added).
   1.3) Push your branch to your remote.

2. Create a fork and a branch of https://github.com/AdoptOpenJDK/openjdk-tests

3. In your branch, point to the change you made in step (1):
   3.1) Open /openjdk-tests/systemtest/build.xml
   3.2) Update the target called clone_openj9-systemtest to include your the name of your branch of your own openj9-systemtest repo's fork, for example:

<target name="clone_openj9-systemtest">
		<exec executable="git" failonerror="false">
			<arg value="clone" />
			<arg value="--depth" />
			<arg value="1" />
			<arg value="-b" />
			<arg value="ibmjvmtimissing" />
			<arg value="https://github.com/Mesbah-Alam/openj9-systemtest.git" />
			<arg value="${SYSTEMTEST_ROOT}/openj9-systemtest" />
		</exec>
	</target>

4. Run in personal build:
   4.1) From a browser, go to ci.adopt test_personal build : https://ci.adoptopenjdk.net/view/work%20in%20progress/job/test_personal/
   4.2) Click "Build with Parameters" and fill them out as follows:
   4.2.1) personalRepo = type in the name of your openjdk-tests fork created in step (2) above.
   4.2.2) personalBranch = type in the name of your openjdk-tests branch created in step (2) above.
   4.2.3) JAVA_IMPL = openj9
   4.2.4 JVM_VERSION = openjdk8-openj9
   4.2.5) TESTPROJECT = systemtest
   4.2.6) TARGET = MauveSingleThreadLoadTest_0
   4.2.7) SDK_RESOURCE = nightly

Leave the rest as default and launch the build. This will download the latest nightly build of openjkd8-openj9 sdk from Adopt and run the build, which will in tern make an attempt to build openj9-systemtest/openj9.test.sharedClasses.jvmti/src/native/sharedClasses.c and reproduce the ibmjvmti.h not found issue.

[pshipton]

@SueChaplain this problem should be added to the 0.8 release notes. The ibmjvmti.h file is missing from the include directory, code that uses "ibm" jvmti extensions can't be compiled.

[Mesbah-Alam]

@pshipton, Hang pointed out that ibmjvmti.h is included in https://github.com/eclipse/openj9/blob/master/runtime/include/ibmjvmti.h. The reason why we are seeing this test failure may have to do with the fact that when compiling the test, we are not supplying the 'runtime' folder into IFLAGS option of the gcc command line of this test's makefile. I am testing out the fix. I will remove this issue from openj9 and create one under openj9-systemtest if the fix works.

[pshipton]

The reported issue is a valid problem. Users shouldn't have to have the OpenJ9 source available in order to use the include file. It should be in the include directory of the "JDK" image produced by an OpenJ9 build, similarly to how jvmi.h is available there.

[Mesbah-Alam]

@hangshao0 FYI.

[jdekonin]

I believe this is a problem with the ibmruntimes/openj9-openjdk-jdk8 extensions wrt 'composition' of the final jdk/jre. If I do a search against Adopt-OpenJ9 built binaries its found for 9 but not 8.

$ ./jdk8u162-b12_openj9-0.8.0/bin/java -version
openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-b12)
Eclipse OpenJ9 VM (build openj9-0.8.0, JRE 1.8.0 Linux amd64-64 Compressed References 20180315_120 (JIT enabled, AOT enabled)
OpenJ9   - e24e8aa9
OMR      - 3e8296b4
JCL      - ee1e77df1d based on jdk8u162-b12)

$ find jdk8u162-b12_openj9-0.8.0/ -name ibmjvmti.h


$ ./jdk-9+181/bin/java -version
openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-adhoc.jenkins.openjdk)
Eclipse OpenJ9 VM (build 2.9, JRE 9 Linux amd64-64 Compressed References 20171027_36 (JIT enabled, AOT enabled)
OpenJ9   - 292f272
OMR      - 9ea665d
OpenJDK  - 640ef65 based on )

$ find jdk-9+181/ -name ibmjvmti.h
jdk-9+181/include/ibmjvmti.h

I think this should be raised against the openj9-openjdk-jdk8 repo.

[babsingh]

I agree with @jdekonin.

openj9-openjdk-jdk9 has openj9-openjdk-jdk9/closed/make/copy/Copy-java.base.gmk
which copies jvmti.h and ibmjvmti.h into the final jdk/jre. Copy-java.base.gmk is called from openj9-openjdk-jdk9/jdk/make/copy/Copy-java.base.gmk in openj9-openjdk-jdk9.

currently, we don't copy ibmjvmti.h in openj9-openjdk-jdk8. in openj9-openjdk-jdk8, we can copy ibmjvmti.h into the final jdk/jre by updating openj9-openjdk-jdk8/closed/make/CopyFiles.gmk. closed/make/CopyFiles.gmk is called from openj9-openjdk-jdk8/jdk/make/CopyFiles.gmk. jdk/make/CopyFiles.gmk already accounts for jvmti.h.

i will create a pull request in openj9-openjdk-jdk8.

[mesbaha]

@babsingh I verified your fix by running the openj9 systemtests sharedClasses jvmti native tests in a personal build using the sdk you provided:

10:39:21 openjdk version "1.8.0_172-internal"
10:39:21 OpenJDK Runtime Environment (build 1.8.0_172-internal-j9build_2018_06_08_23_10-b00)
10:39:21 Eclipse OpenJ9 VM (build master-d8069bf8, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20180608_1398 (JIT enabled, AOT enabled)
10:39:21 OpenJ9 - d8069bf
10:39:21 OMR - ba6e90a
10:39:21 JCL - b96ef997ae based on jdk8u172-b11)