Add JDK11 support and enable in CI #31644

alpar-t · 2018-06-28T07:11:46Z

Update CI to include JDK 11.
mute failing tests with issue ID mentioned
upgrade bouncycastle to avoid invalid manifest errors on JDK 11
improve how BWC builds are logged. This one is not directly related to JDK 11, but helped debug failures while working on it. Each line of the BWC build will be prefixed and indented, to make it easier to distinguish from CI logs that it's the BWC build and not the main Gradle build.

Required to fix `bcprov-jdk15on-1.55.jar; invalid manifest format ` on jdk 11

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

To see if there are other failures on JDK11

Fix JAVA version to 10 for ES 6.3

``` > Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion [bwc] :buildSrc:compileJava [bwc] WARNING: An illegal reflective access operation has occurred [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize() [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [bwc] WARNING: All illegal access operations will be denied in a future release [bwc] :buildSrc:compileGroovy [bwc] :buildSrc:writeVersionProperties [bwc] :buildSrc:processResources [bwc] :buildSrc:classes [bwc] :buildSrc:jar ```

So that we can make sure it's not too new for the build to understand.

elasticmachine · 2018-06-28T07:11:47Z

Pinging @elastic/es-core-infra

closes elastic#31500

alpar-t · 2018-06-28T07:59:02Z

.ci/java-versions.properties

@@ -4,5 +4,5 @@
 # build and test Elasticsearch for this branch. Valid Java versions
 # are 'java' or 'openjdk' followed by the major release number.

-ES_BUILD_JAVA=java10
-ES_RUNTIME_JAVA=java8
+ES_BUILD_JAVA=java11


@jasontedor I'm actually not sure if this is ok, or we should have it only for matrix and do this lather on.

Indeed, we need to hold off on this specific change and leave it to the matrix only for now until we reach conclusion on the minimum runtime JDK for 7.0.0.

alpar-t · 2018-06-29T09:41:30Z

I applied the v6.4.0 label, to back-port this PR -CI changes and muted tests.
Will do it with a separate PR.

nik9000

I hate having all of the awaitsfixes but that is life I guess. Beyond those this looks good to me.

I suppose we assign those failing tests to some owner?

jasontedor · 2018-06-29T19:23:55Z

Can we not awaits fix those and instead make them assumeFalse(JDK 11)?

jasontedor

I left one comment. I am not marking it as request changes because I will not be around next week to approve the change. I would lean on someone else to pick up this thread.

jasontedor · 2018-06-29T19:25:39Z

distribution/bwc/build.gradle

    } else if ("6.2".equals(bwcBranch)) {
      environment('JAVA_HOME', getJavaHome(it, 9))
+      environment('RUNTIME_JAVA_HOME', getJavaHome(it, 9))


What is the reason for this change? I think it is because we can't run these builds when the run-time Java is JDK 11 and that is fine. If that's the case I think that we should lock these to the minimum supported JDK which is JDK 8.

jasontedor · 2018-06-29T19:26:54Z

...t-attachment/src/test/java/org/elasticsearch/ingest/attachment/AttachmentProcessorTests.java

@@ -295,6 +295,7 @@ public void testNonExistentWithoutIgnoreMissing() throws Exception {
        return attachmentData;
    }

+    @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/31305")


I think that we should use assumeFalse(JDK 11) on these (use the JavaVersion.current().compareTo(JavaVersion.parse("11")) that I pointed to before).

jasontedor · 2018-06-29T19:27:01Z

plugins/repository-hdfs/src/test/java/org/elasticsearch/repositories/hdfs/HdfsTests.java

@@ -43,6 +43,7 @@
        return pluginList(HdfsPlugin.class);
    }

+    @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/31498")


I think that we should use assumeFalse(JDK 11) on these (use the JavaVersion.current().compareTo(JavaVersion.parse("11")) that I pointed to before).

rjernst

This looks good. I left a few comments. Additionally, I think any issues that are linked here as TODOs need to be made blockers so that they are at least investigated and their impact assessed before 6.4.0.

rjernst · 2018-07-02T18:25:50Z

buildSrc/src/main/resources/forbidden/jdk-signatures.txt

@@ -88,7 +88,6 @@ java.lang.Thread#getAllStackTraces()

 @defaultMessage Stopping threads explicitly leads to inconsistent states. Use interrupt() instead.
 java.lang.Thread#stop()
-java.lang.Thread#stop(java.lang.Throwable)


Why is this removed from forbidden apis? The forbidden signatures should against our minimum runtime, which is still java 8.

The method was removed in jdk 11 and causing the test to fail.
Do we have per JDK version forbidden APIs ?

We do not. I think this is related to running forbidden apis in a separate jvm. We should always be running it with the minimum supported version.

Can you please comment this out and reference the issue to run forbidden apis in a forked jvm.

Done, will remove comment once #31715 is fixed.

If this issue was the only reason to no longer use the gradle plugin and fork instead, I can just say: The above forbidden-api signature can be removed anyways from elasticsearch (together with the previous one). Forbiddenapis has those signatures already in its "jdk-deprecated" ones, as it's a known problem - so there is no need to have it in Elasticsearch's custom signatures. "jdk-deprecated" automatically handles deprecations that were removed.

IMHO, #31715 was not a good idea, sorry!

https://github.com/policeman-tools/forbidden-apis/blob/master/src/main/resources/de/thetaphi/forbiddenapis/signatures/jdk-deprecated-1.8.txt#L175-L180

and

https://github.com/policeman-tools/forbidden-apis/blob/master/src/main/resources/de/thetaphi/forbiddenapis/signatures/jdk-deprecated-1.8.txt#L7

rjernst · 2018-07-02T18:26:45Z

distribution/bwc/build.gradle

@@ -196,3 +204,25 @@ subprojects {
    }
  }
 }
+
+class IdentingOutputStream extends OutputStream {


typo: Identing -> Indenting

rjernst · 2018-07-02T18:33:02Z

distribution/bwc/build.gradle

+  public void write(int[] bytes, int offset, int length) {
+    delegate.write(
+            new String(bytes, offset, length)
+                    .replace("\n", "\n  [bwc] ")


I think this is going to be pretty inefficient. Instead, you can loop over the bytes looking for newline. When you find a newline, emit the array up to that point. Then emit a fixed byte array you have with the newline and prefix. Then continue the loop. You won't need to create any temporary objects.

* Upgrade bouncycastle Required to fix `bcprov-jdk15on-1.55.jar; invalid manifest format ` on jdk 11 * Downgrade bouncycastle to avoid invalid manifest * Add checksum for new jars * Update tika permissions for jdk 11 * Mute test failing on jdk 11 * Add JDK11 to CI * Thread#stop(Throwable) was removed http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html * Disable failing tests #31456 * Temprorarily disable doc tests To see if there are other failures on JDK11 * Only blacklist specific doc tests * Disable only failing tests in ingest attachment plugin * Mute failing HDFS tests #31498 * Mute failing lang-painless tests #31500 * Fix backwards compatability builds Fix JAVA version to 10 for ES 6.3 * Add 6.x to bwx -> java10 * Prefix out and err from buildBwcVersion for readability ``` > Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion [bwc] :buildSrc:compileJava [bwc] WARNING: An illegal reflective access operation has occurred [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize() [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [bwc] WARNING: All illegal access operations will be denied in a future release [bwc] :buildSrc:compileGroovy [bwc] :buildSrc:writeVersionProperties [bwc] :buildSrc:processResources [bwc] :buildSrc:classes [bwc] :buildSrc:jar ``` * Also set RUNTIME_JAVA_HOME for bwcBuild So that we can make sure it's not too new for the build to understand. * Align bouncycastle dependency * fix painles array tets closes #31500 * Update jar checksums * Keep 8/10 runtime/compile untill consensus builds on 11 * Only skip failing tests if running on Java 11 * Failures are dependent of compile java version not runtime * Condition doc test exceptions on compiler java version as well * Disable hdfs tests based on runtime java * Set runtime java to minimum supported for bwc * PR review * Add comment with ticket for forbidden apis

* 6.x: Test: Do not remove xpack templates when cleaning (#31642) SQL: Allow long literals (#31777) SQL: Fix incorrect message for aliases (#31792) Detach Transport from TransportService (#31727) 6.3.1 release notes (#31829) Add unreleased version 6.3.2 [ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817) [ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800) [ML] Fix calendar and filter updates from non-master nodes (#31804) Fix license header generation on Windows (#31790) mark XPackRestIT.test {p0=monitoring/bulk/10_basic/Bulk indexing of monitoring data} as AwaitsFix Add JDK11 support without enabling in CI (#31644) Watcher: Fix check for currently executed watches (#31137) [DOCS] Fixes 6.3.0 release notes (#31771) Watcher: Ensure correct method is used to read secure settings (#31753) [ML] Rate limit established model memory updates (#31768) SQL: Update CLI logo

* master: REST high-level client: add get index API (#31703) SQL: Allow long literals (#31777) SQL: Fix incorrect message for aliases (#31792) Test: Do not remove xpack templates when cleaning (#31642) Reduce more raw types warnings (#31780) Add unreleased version 6.3.2 Scripting: Remove support for deprecated StoredScript contexts (#31394) [ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817) [ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800) [ML] Fix calendar and filter updates from non-master nodes (#31804) Fix license header generation on Windows (#31790) mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix Correct exclusion of test on JDK 11 Fix doclint jdk 11 Add JDK11 support and enable in CI (#31644) Watcher: Fix check for currently executed watches (#31137) Watcher: Ensure correct method is used to read secure settings (#31753) SQL: Update CLI logo

alpar-t added 18 commits June 28, 2018 09:19

Upgrade bouncycastle

d26bf2a

Required to fix `bcprov-jdk15on-1.55.jar; invalid manifest format ` on jdk 11

Downgrade bouncycastle to avoid invalid manifest

e132892

Add checksum for new jars

efe1844

Update tika permissions for jdk 11

3acd7b5

Mute test failing on jdk 11

283b708

Thread#stop(Throwable) was removed

6091a3f

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

Add JDK11 to CI

0060bb7

Disable failing tests elastic#31456

e433bce

Temprorarily disable doc tests

be6fe54

To see if there are other failures on JDK11

Only blacklist specific doc tests

dbe3a08

Disable only failing tests in ingest attachment plugin

6e10477

Mute failing HDFS tests elastic#31498

aae1184

Mute failing lang-painless tests elastic#31500

d7e299e

Fix backwards compatability builds

a7a49db

Fix JAVA version to 10 for ES 6.3

Add 6.x to bwx -> java10

da094bc

Also set RUNTIME_JAVA_HOME for bwcBuild

00d3b7e

So that we can make sure it's not too new for the build to understand.

Align bouncycastle dependency

1c19cf8

alpar-t added :Delivery/Build Build or test infrastructure jdk11 labels Jun 28, 2018

alpar-t added 2 commits June 28, 2018 10:32

fix painles array tets

ef58e75

closes elastic#31500

Update jar checksums

b744d1e

alpar-t commented Jun 28, 2018

View reviewed changes

Keep 8/10 runtime/compile untill consensus builds on 11

ba87083

alpar-t added v7.0.0 v6.4.0 labels Jun 29, 2018

nik9000 reviewed Jun 29, 2018

View reviewed changes

jasontedor reviewed Jun 29, 2018

View reviewed changes

alpar-t added 6 commits July 2, 2018 14:26

Only skip failing tests if running on Java 11

fb05672

Failures are dependent of compile java version not runtime

cfc5923

Merge remote-tracking branch 'origin/master' into feature/jdk11-31230

9961d8e

Condition doc test exceptions on compiler java version as well

e6720b8

Disable hdfs tests based on runtime java

41d97c1

Set runtime java to minimum supported for bwc

188460a

rjernst reviewed Jul 2, 2018

View reviewed changes

alpar-t added 3 commits July 3, 2018 18:12

PR review

6d9fb3f

Add comment with ticket for forbidden apis

13f536b

Merge commit 'e9f8442' into feature/jdk11-31230

062561c

alpar-t merged commit cf2295b into elastic:master Jul 5, 2018

alpar-t deleted the feature/jdk11-31230 branch July 5, 2018 03:24

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add JDK11 support and enable in CI #31644

Add JDK11 support and enable in CI #31644

alpar-t commented Jun 28, 2018 •

edited

Loading

elasticmachine commented Jun 28, 2018

alpar-t Jun 28, 2018

jasontedor Jun 28, 2018

alpar-t commented Jun 29, 2018

nik9000 left a comment

jasontedor commented Jun 29, 2018

jasontedor left a comment

jasontedor Jun 29, 2018

jasontedor Jun 29, 2018

jasontedor Jun 29, 2018

rjernst left a comment

rjernst Jul 2, 2018

alpar-t Jul 3, 2018

rjernst Jul 3, 2018

rjernst Jul 3, 2018

alpar-t Jul 4, 2018

uschindler Sep 17, 2018

uschindler Sep 17, 2018

rjernst Jul 2, 2018

rjernst Jul 2, 2018

Add JDK11 support and enable in CI #31644

Add JDK11 support and enable in CI #31644

Conversation

alpar-t commented Jun 28, 2018 • edited Loading

elasticmachine commented Jun 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpar-t commented Jun 29, 2018

nik9000 left a comment

Choose a reason for hiding this comment

jasontedor commented Jun 29, 2018

jasontedor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rjernst left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpar-t commented Jun 28, 2018 •

edited

Loading