Create Dockerfiles that allow test execution #23

smlambert · 2017-09-14T21:36:35Z

Basing from those Dockerfiles that build openj9 (https://github.com/eclipse/openj9/tree/master/buildenv/docker), we should have matching set to allow test execution.

There are a few additional prereqs needed for test (see https://github.com/eclipse/openj9/blob/master/test/README.md for details on testing openj9).

terryzuoty · 2017-09-14T21:55:33Z

The dockerfile in link above doesn’t seem to build JDK or get a JDK, it simply setting up the environment. Maybe I am wrong if I miss something?

Do we want to integrate the test execution into the same dockerfile or have another dockerfile to execute tests?

smlambert · 2017-09-15T02:53:56Z

Separate dockerfiles for test, can be put in buildenv/docker/openj9test (or something similar that shadow existing ones). See the website instructions for the expected usage of the existing files, https://www.eclipse.org/openj9/oj9_build.html

DanHeidinga · 2017-09-15T03:17:40Z

+1 to putting the test-related docker files in the buildenv/docker directory. Does the test subdirectory need an openj9 prefix there or is test sufficient?

smlambert · 2017-09-15T15:15:57Z

buildenv/docker/test sounds good

we should update the webpage with additional information on these new dockerfiles once they are ready (as the instructions read right now, they imply that you would be able to move on the test step from using the existing dockerfiles, which is not the case).

jdekonin · 2017-09-15T15:27:07Z

Why would we not just add the additional requirements to the existing dockerfile(s)? or at a minimum built on top of the initial dockerfile?

smlambert · 2017-09-15T15:35:48Z

"Basing from those Dockerfiles that build openj9"

mleipe · 2017-09-15T18:03:57Z

Should we put the OpenJ9 Docker images onto Dockerhub such that the test Dockerfiles can be FROM them?

ghost · 2017-09-17T01:30:31Z

@mleipe #38

Yes, we shall put openj9 image on hub.docker.com

terryzuoty · 2017-09-18T21:39:31Z

Taking a second look at the Dockfiles in openj9/buildenv/docker/jdk9/<platform>/ubuntu16, I found that most of the executable dependencies have already been included in the Dockerfiles, only ant and ant-contrib are missing for test purpose.

After trying the whole building and testing process within Docker container, I think we need two separate tasks to solve two different Docker-related user scenarios.

Scenario 1: I want to run OpenJ9 tests against a downloaded OpenJ9 binary (OpenJ9 SDK). Or I just want to test my new test codes against a pre-built OpenJ9 SDK.

In this case, we just need a light weight Docker image which contains minimum dependencies for executing tests against OpenJ9 SDK. So Task 1 is to create a Dockerfile which should be independent from those Dockerfiles which build OpenJ9. It should be located in open9/buildenv/docker/test as Shelley and Dan mentioned.

Scenario 2: I want to build the OpenJ9 and also test it.

In this case, it will be convenient to build OpenJ9 and run tests in the same Docker image. So Task 2 is adding ant and ant-contrib into the Dockerfiles.
After adding ant and ant-contrib into the Dockerfiles and building it, (openj9/buildenv/docker/jdk9/<platform>/ubuntu16), the processing time of building Docker image increased less than 1 minutes.

In the contrast, building OpenJ9 and running tests in two different Docker images separate the clone-clone-make story, which we want to avoid. Also, the instructions in https://www.eclipse.org/openj9/oj9_build.html make it very difficult to build OpenJ9 and run tests in two separate Docker images. The openj9-openjdk-jdk9 repo is cloned within the Docker container which makes the OpenJ9 SDK built in this Docker image difficult to shared outside.

Summary:
a) we need to create an independent test-only Dockerfile for testing purpose
b) we also need to update those Dockerfiles which build the OpenJ9 to make them available to run tests in the same Docker image.
c) I also think we should update the instruction in https://www.eclipse.org/openj9/oj9_build.html. The openj9-openjdk-jdk9 repo should be cloned outside of Docker and mounted to a Docker container when needed. (https://forums.docker.com/t/best-practices-for-getting-code-into-a-container-git-clone-vs-copy-vs-data-container/4077 and http://blog.cloud66.com/how-to-get-code-into-a-docker-container/).

terryzuoty · 2017-09-19T22:40:15Z

For Scenario2, it seems we have to separate two different Dockerfile and use two different Docker images to build OpenJ9 and run tests.

Test Dockerfile and build Dockerfile cannot be combined due to:

executable dependencies conflicts
existing of build-essential in OS will fail the OpenJ9 compilation
lack of build-essential will fail getting perl module

In this result, I think should update the instruction in https://www.eclipse.org/openj9/oj9_build.html. The openj9-openjdk-jdk9 repo should be cloned outside of Docker and mounted to a Docker container when needed. Then the build OpenJ9 SDK can be mounted to the other testing Docker image

mgaudet · 2017-09-20T11:11:20Z

build-essential is a meta package, that describes a whole bunch of dependencies -- we could eliminate it, and expand the dependencies explicitly to support build+test, though, I'm curious about executable-dependencies conflicts: What conflicts do you see?

terryzuoty · 2017-09-20T14:51:15Z

Take this Dockerfile as an example:

FROM ubuntu:16.04

# Install required OS tools
RUN apt-get update \
  && apt-get install -qq -y --no-install-recommends \
    autoconf \
    ca-certificates \
    ccache \
    cpio \
    file \
    g++-4.8 \
    gcc-4.8 \
    git \
    git-core \
    libasound2-dev \
    libcups2-dev \
    libelf-dev \
    libfreetype6-dev \
    libnuma-dev \
    libx11-dev \
    libxext-dev \
    libxrender-dev \
    libxt-dev \
    libxtst-dev \
    make \
    openjdk-8-jdk \
    pkg-config \
    realpath \
    ssh \
    unzip \
    wget \
    zip \
  && rm -rf /var/lib/apt/lists/*

# Create links for c++,g++,cc,gcc
RUN ln -s g++ /usr/bin/c++ \
  && ln -s g++-4.8 /usr/bin/g++ \
  && ln -s gcc /usr/bin/cc \
  && ln -s gcc-4.8 /usr/bin/gcc

# Download and setup freemarker.jar to /root/freemarker.jar
RUN cd /root \
  && wget https://sourceforge.net/projects/freemarker/files/freemarker/2.3.8/freemarker-2.3.8.tar.gz/download -O freemarker.tgz \
  && tar -xzf freemarker.tgz freemarker-2.3.8/lib/freemarker.jar --strip=2 \
  && rm -f freemarker.tgz

# Install required executable dependencies for testing
RUN apt-get update \
    && apt-get install -qq -y --no-install-recommends \
    ant \
    ant-contrib \
    perl \
    vim \
  && rm -rf /var/lib/apt/lists/*

# Install Docker module to run test framework
RUN echo yes | cpan install JSON
RUN echo yes | cpan install Text::CSV

WORKDIR /root

Error messages are different.

Without build-essential, the process of installing perl modules will fail with message:

"/usr/bin/perl" "/usr/share/perl/5.22/ExtUtils/xsubpp"  -typemap "/usr/share/perl/5.22/ExtUtils/typemap" -typemap "typemap"  XS.xs > XS.xsc && mv XS.xsc XS.c
x86_64-linux-gnu-gcc -c   -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g   -DVERSION=\"3.04\" -DXS_VERSION=\"3.04\" -fPIC "-I/usr/lib/x86_64-linux-gnu/perl/5.22/CORE"   XS.c
/bin/sh: 1: x86_64-linux-gnu-gcc: not found
Makefile:335: recipe for target 'XS.o' failed
make: *** [XS.o] Error 127
  MLEHMANN/JSON-XS-3.04.tar.gz
  /usr/bin/make -- NOT OK
The command '/bin/sh -c echo yes | cpan install JSON' returned a non-zero code: 1

if we put build-essential in front of gcc-4.8 and g++-4.8, installation process will give error message like:

 ---> 3eb042f87d55
Removing intermediate container 46abb3b8ae0a
Step 3/8 : RUN ln -s g++ /usr/bin/c++   && ln -s g++-4.8 /usr/bin/g++   && ln -s gcc /usr/bin/cc   && ln -s gcc-4.8 /usr/bin/gcc
 ---> Running in d8d24b84a82f
ln: failed to create symbolic link '/usr/bin/c++': File exists
The command '/bin/sh -c ln -s g++ /usr/bin/c++   && ln -s g++-4.8 /usr/bin/g++   && ln -s gcc /usr/bin/cc   && ln -s gcc-4.8 /usr/bin/gcc' returned a non-zero code: 1

If build-essential is installed during setting up the test environment (put behind gcc-4.8 and g++-4.8), the Docker image building process can pass, but the compilation of OpenJ9 in the Docker container will failed like:

../tr.source/trj9/runtime/MethodMetaData.h:289:43: error: inline function 'get32BitNextExceptionTableEntryFSD' declared but never defined [-Werror]
 JITINLINE J9JIT32BitExceptionTableEntry * get32BitNextExceptionTableEntryFSD(J9JIT32BitExceptionTableEntry* handlerCursor, UDATA fullSpeedDebug);
                                           ^
../tr.source/trj9/runtime/MethodMetaData.h:288:43: error: inline function 'get32BitNextExceptionTableEntry' declared but never defined [-Werror]
 JITINLINE J9JIT32BitExceptionTableEntry * get32BitNextExceptionTableEntry(J9JIT32BitExceptionTableEntry* handlerCursor);
                                           ^
../tr.source/trj9/runtime/MethodMetaData.h:287:43: error: inline function 'get16BitFirstExceptionDataField' declared but never defined [-Werror]
 JITINLINE J9JIT16BitExceptionTableEntry * get16BitFirstExceptionDataField(J9TR_MethodMetaData* metaData);
                                           ^
../tr.source/trj9/runtime/MethodMetaData.h:286:43: error: inline function 'get32BitFirstExceptionDataField' declared but never defined [-Werror]
 JITINLINE J9JIT32BitExceptionTableEntry * get32BitFirstExceptionDataField(J9TR_MethodMetaData* metaData);
                                           ^
mv -f xnathelp.s xnathelp.hold
cc1: all warnings being treated as errors
../makelib/targets.mk:294: recipe for target 'jswalk.o' failed
make[5]: *** [jswalk.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[5]: Leaving directory '/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/vm/codert_vm'
makefile:1111: recipe for target 'j9codert_vm' failed
make[4]: *** [j9codert_vm] Error 2
make[4]: Leaving directory '/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/vm'
OpenJ9.gmk:216: recipe for target 'build-j9' failed
make[3]: *** [build-j9] Error 1
/openj9-openjdk-jdk9/closed/make/Main.gmk:30: recipe for target 'j9vm-build' failed
make[2]: *** [j9vm-build] Error 1
make[2]: *** Waiting for unfinished jobs....

ERROR: Build failed for target 'all' in configuration 'linux-x86_64-normal-server-release' (exit code 2) 

=== Make failed targets repeated here ===
../makelib/targets.mk:294: recipe for target 'jswalk.o' failed
makefile:1111: recipe for target 'j9codert_vm' failed
OpenJ9.gmk:216: recipe for target 'build-j9' failed
/openj9-openjdk-jdk9/closed/make/Main.gmk:30: recipe for target 'j9vm-build' failed
=== End of repeated output ===

Hint: Try searching the build log for the name of the first failed target.
Hint: See common/doc/building.html#troubleshooting for assistance.

/openj9-openjdk-jdk9/make/Init.gmk:291: recipe for target 'main' failed
make[1]: *** [main] Error 1
/openj9-openjdk-jdk9/make/Init.gmk:185: recipe for target 'all' failed
make: *** [all] Error 2

terryzuoty · 2017-09-20T17:50:01Z

Inspired by Mathew's comment, found out that it is the gcc-5 and g++-5 included inbuild-essential causing the conflict. After installing build-essential, /usr/bin/gcc and /usr/bin/g++ was overwritten to g++5 and gcc5, which wasn't compatible with OpenJ9 compilation.

Update the Pull Request changeset to reset the symlink of /usr/bin/gcc and /usr/bin/g++ to gcc4.8 and g++4.8, then OpenJ9 build and test execution can be intergrated in the same Docker image.

mgaudet · 2017-09-20T18:03:17Z

(opened #88, as I suspect if we used update-alternatives, this could be avoided)

mstoodle · 2017-09-20T18:16:33Z

@mleipe @netroby this project does not create binaries, so we should not be uploading anything to Dockerhub, please.

mstoodle · 2017-09-20T18:18:25Z

Note that FROM can refer to images on your local machine, so you can build one and then build the other from it (this is how the AdoptOpenJDK docker files work, see adoptium/temurin-build#166 ).

Of course, that assumes you can make it work as "build one" and then "extend that one with additional things" :) .

mleipe · 2017-09-20T18:21:42Z

Thanks for the clarification, @mstoodle !

mstoodle · 2017-09-20T18:23:23Z

For my vote, I would actually prefer this to be done via a separate Dockerfile which builds on the root "build" one. That way you should be able to run tests on an image that can also (re)build OpenJDK with OpenJ9 and will automatically pick up any changes if we need to update that base recipe. Maybe that suggests the two files should be in the same directory?

* create a Dockerfile to allow test execution and build OpenJ9 * test Dockerfile is located in buildenv/docker/test '[skip ci]' Issue: eclipse-openj9#23 Signed-off-by: Tianyu Zuo <[email protected]>

terryzuoty · 2017-09-22T00:51:53Z

Issue solved in #79

* Register Class into Table (eclipse-openj9#23) - register performed in createramclass - added register functions into internalfuncs - changed naming scheme Signed-off-by: akshayben <[email protected]>

…winstance Check for array elements with value type fields

smlambert added the enhancement label Sep 14, 2017

terryzuoty mentioned this issue Sep 19, 2017

Create Dockerfiles that allow test execution #79

Merged

mgaudet mentioned this issue Sep 20, 2017

Use update-alternatives in docker files rather than straight symlinks #88

Open

smlambert added the comp:test label Sep 21, 2017

tajila mentioned this issue Sep 22, 2017

Crash running gradle reported on StackOverflow #110

Closed

smlambert closed this as completed Sep 26, 2017

urisimchoni mentioned this issue Apr 2, 2018

JIT failure when JVMTI field watches are added #1572

Closed

DanHeidinga mentioned this issue Jun 3, 2019

jdk11: jtreg java/util/concurrent/ArrayBlockingQueue/WhiteBox.java crashes on MacOS #5988

Closed

kusumachalasani mentioned this issue Oct 14, 2019

Dacapo sub-benchmark "eclipse" crashes when run with large load #7452

Open

hzongaro pushed a commit to hzongaro/openj9 that referenced this issue Apr 6, 2020

Merge pull request eclipse-openj9#23 from hzongaro/valuetype-array-ne…

e8a8841

…winstance Check for array elements with value type fields

mpirvu mentioned this issue Apr 19, 2021

"Unknown bytecode to JIT 206" assert failure during testJITServer_1 #12468

Closed

manqingl mentioned this issue Jun 16, 2021

Infinite loop leading to memory corruption and crash. #12979

Closed

a0304 mentioned this issue Aug 3, 2021

OpenJ9 JVM crash when loading native library in Linux #13269

Open

urisimchoni mentioned this issue Nov 23, 2021

Crash or OutOfMemoryError when using AsyncGetCallTrace() #13838

Closed

mkarrahcl mentioned this issue Sep 26, 2023

"dump java core" is hanging process on RHEL 9.1/9.2 using Semeru OpenJDK 17 #18204

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Dockerfiles that allow test execution #23

Create Dockerfiles that allow test execution #23

smlambert commented Sep 14, 2017

terryzuoty commented Sep 14, 2017 •

edited

Loading

smlambert commented Sep 15, 2017

DanHeidinga commented Sep 15, 2017

smlambert commented Sep 15, 2017

jdekonin commented Sep 15, 2017

smlambert commented Sep 15, 2017

mleipe commented Sep 15, 2017

ghost commented Sep 17, 2017

terryzuoty commented Sep 18, 2017 •

edited

Loading

terryzuoty commented Sep 19, 2017 •

edited

Loading

mgaudet commented Sep 20, 2017

terryzuoty commented Sep 20, 2017 •

edited

Loading

terryzuoty commented Sep 20, 2017

mgaudet commented Sep 20, 2017

mstoodle commented Sep 20, 2017

mstoodle commented Sep 20, 2017 •

edited

Loading

mleipe commented Sep 20, 2017

mstoodle commented Sep 20, 2017 •

edited

Loading

terryzuoty commented Sep 22, 2017

Create Dockerfiles that allow test execution #23

Create Dockerfiles that allow test execution #23

Comments

smlambert commented Sep 14, 2017

terryzuoty commented Sep 14, 2017 • edited Loading

smlambert commented Sep 15, 2017

DanHeidinga commented Sep 15, 2017

smlambert commented Sep 15, 2017

jdekonin commented Sep 15, 2017

smlambert commented Sep 15, 2017

mleipe commented Sep 15, 2017

ghost commented Sep 17, 2017

terryzuoty commented Sep 18, 2017 • edited Loading

terryzuoty commented Sep 19, 2017 • edited Loading

mgaudet commented Sep 20, 2017

terryzuoty commented Sep 20, 2017 • edited Loading

terryzuoty commented Sep 20, 2017

mgaudet commented Sep 20, 2017

mstoodle commented Sep 20, 2017

mstoodle commented Sep 20, 2017 • edited Loading

mleipe commented Sep 20, 2017

mstoodle commented Sep 20, 2017 • edited Loading

terryzuoty commented Sep 22, 2017

terryzuoty commented Sep 14, 2017 •

edited

Loading

terryzuoty commented Sep 18, 2017 •

edited

Loading

terryzuoty commented Sep 19, 2017 •

edited

Loading

terryzuoty commented Sep 20, 2017 •

edited

Loading

mstoodle commented Sep 20, 2017 •

edited

Loading

mstoodle commented Sep 20, 2017 •

edited

Loading