JDK8 OpenJ9 Gives Different Results from HotSpot and Android Runtime #15306

connglli · 2022-06-13T15:41:09Z

Java version

openjdk version "1.8.0_342-internal"
OpenJDK Runtime Environment (build 1.8.0_342-internal-_2022_06_10_15_18-b00)
Eclipse OpenJ9 VM (build master-3d06b2f9c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20220610_000000 (JIT enabled, AOT enabled)
OpenJ9   - 3d06b2f9c
OMR      - cf8ddbd1a
JCL      - 2bb179375a based on jdk8u342-b05)

Javac version

javac 1.8.0_342-internal

Summary of problem

The following test makes OpenJ9 (with above version) to produce different result every time run.

class Test {
  int N;
  long instanceCount;
  int iFld;
  float fFld;
  long vMeth_check_sum;

  void vMeth(long l1, short s2, double d1) {
    int i6, i8, i10, i11 = 12, i12 = 210;
    for (i6 = 7; i6 < 149; ++i6)
      for (i8 = 1; i8 < 4; i8++)
        for (i10 = 1; i10 < 2; ++i10) {
          i11 += i12;
          iFld = i10;
        }
    vMeth_check_sum += i11;
  }

  int iMeth1(boolean b1, float f1, int i5) {
    float fArr1[] = new float[N];
    for (int ax$11 = 0; ax$11 < 6361; ax$11 += 1)
      vMeth(5063039435690207991L, (short) 1019863587, 0.2494915619941982);
    long meth_res = Double.doubleToLongBits(checkSum(fArr1));
    return (int) meth_res;
  }

  int iMeth(long l, short s1, float f) {
    int i1, i4 = 11;
    boolean b = true;
    byte[] byArr = new byte[N];
    for (i1 = 11; i1 < 212; i1++) i4 *= iMeth1(b, 1.457F, i1);
    long meth_res = checkSum(byArr);
    return (int) meth_res;
  }

  void mainTest(String[] strArr1) {
    short s = 20160;
    s = (short) iMeth(instanceCount, s, fFld);
    System.out.println(vMeth_check_sum);
  }

  public static void main(String[] strArr) {
    Test _instance = new Test();
    _instance.mainTest(strArr);
  }

  public static long checkSum(byte[] a) {
    long sum = 0;
    for (int j = 0; j < a.length; j++) {
      sum += (byte) (a[j] / (j + 1) + a[j] % (j + 1));
    }
    return sum;
  }

  public static double checkSum(float[] a) {
    double sum = 0;
    for (int j = 0; j < a.length; j++) {
      sum += (a[j] / (j + 1) + a[j] % (j + 1));
    }
    return sum;
  }
}

Results of OpenJ9: every time you run it, OpenJ9 gives a different result:

$ ./OpenJ9/jdk8/bin/java Test
70388485152
$ ./OpenJ9/jdk8/bin/java Test
66974154792
$ ./OpenJ9/jdk8/bin/java Test
80631476232

However, HotSpot (JDK8/11/17) and the Android Runtime (ART) are always giving a deterministic and the same result:

$ ./HotSpot/jdk8u/bin/java Test
114395409792
$ ./HotSpot/jdk11u/bin/java Test
114395409792
$ ./HotSpot/jdk17u/bin/java Test
114395409792
$ ./Android/host-art/host/linux-x86/bin/art --no-compile --64 -- -cp test.jar Test
114395409792

The text was updated successfully, but these errors were encountered:

pshipton · 2022-06-13T16:35:50Z

When I try it I often get the correct answer, but occasionally I see a wrong answer.
It also occurs on the last release jdk8u332 - 0.32, and on jdk8u322 - 0.30. I didn't try earlier versions.
Running with -Xint produces the correct answer, I'll start by assuming this is a JIT issue.

The following runs it in a loop so it's easier to reproduce the failures.

import java.io.*;

public class Run {
public static void main(String[] args) throws Exception {
	long count = 0;
	while(true) {
		ProcessBuilder pb = new ProcessBuilder(args);
		Process proc = pb.start();
		InputStream stream = proc.getInputStream();
		BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
		String line = reader.readLine();
		if (!line.equals("114395409792")) {
			System.out.println("Iteration " + count + ": " + line);
		}
		count += 1;
	}
}
}

pshipton · 2022-06-13T16:36:42Z

This seems a serious problem so I'm starting with a blocker status for the next release.

@0xdaryl pls take a look.

pshipton · 2022-06-13T16:52:40Z

I also tried it on plinux (11.0.14.1) where it was (almost) always giving the wrong answer.

pshipton · 2022-06-13T17:23:11Z

I found that on plinux, the problem still reproduces with a smaller case. Even on xlinux you can remove the checkSum() of the arrays and still reproduce it.

class TestCalc {
  long vMeth_check_sum;

  void vMeth() {
    int i6, i8, i10, i11 = 12, i12 = 210;
    for (i6 = 7; i6 < 149; ++i6)
      for (i8 = 1; i8 < 4; i8++)
        for (i10 = 1; i10 < 2; ++i10) {
          i11 += i12;
        }
    vMeth_check_sum += i11;
  }

  int iMeth1() {
    for (int ax$11 = 0; ax$11 < 6361; ax$11 += 1)
      vMeth();
    return 0;
  }

  int iMeth() {
    int i1, i4 = 11;
    for (i1 = 11; i1 < 212; i1++) i4 *= iMeth1();
    return 0;
  }

  void mainTest(String[] strArr1) {
    iMeth();
    System.out.println(vMeth_check_sum);
  }

  public static void main(String[] strArr) {
    TestCalc _instance = new TestCalc();
    _instance.mainTest(strArr);
  }
}

pshipton · 2022-06-13T17:56:40Z

You can change the loop in iMeth() or the outer loop in vMeth(), but if the inner loops in vMeth() are modified to start from zero, then the problem stops occurring.

0xdaryl · 2022-06-14T14:09:51Z

Seems to occur when Test.iMeth1() is compiled at scorching. ExpressionsSimplification may be the culprit.

connglli · 2022-06-15T19:54:57Z

@pshipton

Could you please check if the following Test.java is a duplicate of this issue? Though they looks quite different, they have similar symptoms to each other, i.e., they give different results every time you run it.

If it is indeed a duplicate, just skip it and I won't report other tests in our hands that behave similar symptoms; otherwise, I will start a new issue thread for you guys.

class Test {
  final int N = 256;
  long instanceCount;
  float fFld;
  int iFld;
  short sFld;
  long vMeth_check_sum;

  void vMeth(int i2) {
    int i17 = -244, i19 = 23560, iArr[] = new int[N], iArr2[][] = new int[N][N];
    init(iArr2, 11);
    for (int i3 : iArr) {
      int i16;
      i3 = (int) instanceCount++;
      i2 *= instanceCount;
      switch ((i2 >>> 1) % 6 + 6) {
        case 6:
          for (i17 = 2; i17 > 1; i17 -= 3) i16 = (int) instanceCount;
        case 7:
          iArr2[(i17 >>> 1) % N][i19 % N] *= fFld;
        case 8:
          break;
        case 10:
          vMeth_check_sum += checkSum(iArr2);
      }
    }
  }

  void vSmallMeth(int i, short s, int i1) {
    vMeth(i);
  }

  void mainTest(String[] strArr1) {
    for (int smallinvoc = 0; smallinvoc < 238; smallinvoc++) vSmallMeth(25982, sFld, iFld);
    System.out.println(vMeth_check_sum);
  }

  public static void main(String[] strArr) {
    boolean ax$16;
    Test _instance = new Test();
    _instance.mainTest(strArr);
  }

  public static void init(int[][] a, int seed) {
    for (int j = 0; j < a.length; j++) {
      init(a[j], seed);
    }
  }

  public static void init(int[] a, int seed) {
    for (int j = 0; j < a.length; j++) {
      a[j] = (j % 2 == 0) ? seed + j : seed - j;
    }
  }

  public static long checkSum(int[] a) {
    long sum = 0;
    for (int j = 0; j < a.length; j++) {
      sum += (a[j] / (j + 1) + a[j] % (j + 1));
    }
    return sum;
  }

  public static long checkSum(int[][] a) {
    long sum = 0;
    for (int j = 0; j < a.length; j++) {
      sum += checkSum(a[j]);
    }
    return sum;
  }
}

FYI. The result of HotSpot JDK8/11/17 and the Android Runtime is -7725525924.

Great thanks that for your patience on validating our bug reports. You guys are one of the most active and optimistic developers I've met. :-)

pshipton · 2022-06-15T19:56:09Z

I assume @0xdaryl or a delegate will do that investigation.

connglli · 2022-06-15T19:59:26Z

Thanks for you patience and enthusiasm @pshipton, as well as @0xdaryl.

pshipton · 2022-06-15T20:22:24Z

We have questions about using the test cases you've provided in our regression test suite. Are there any existing licenses or concerns about putting the test cases under the Eclipse v2 license? @llxia is interested in having you open a Pull Request to deliver the test cases, if you are are agreeable to that. We'd try to make it easy so you don't need to be concerned much about test frameworks. You'd have to sign the Eclipse ECA agreement for that, which is the typical requirement for delivering code.
See https://github.com/eclipse-openj9/openj9/blob/master/CONTRIBUTING.md

connglli · 2022-06-16T10:55:44Z

As for the first question, we're working on an automated testing tool based on many existing fuzzers; but it hasn't been released yet. Currently, tests that we have reported is a hybrid of our tool and a fuzzer namely Java*Fuzzer which is licensed by Apache V2. I'm not quite familiar with the licenses, but as far as I know Apache V2 grants you the permission to adapt/modify their code. For ours, you are free to use them. Therefore, I think it's sufficient that you add the Java*Fuzzer's license when including then in your regression test suites.

As for the second question, sorry that currently we don't have that time to make formal contributions using PR following some guidelines. Issuetracker is much more convenient to us at present. But we could consider that when we are free.

A big thanks for your invitation.

pshipton · 2022-06-16T14:28:48Z

Sounds good, thanks. Eclipse OpenJ9 is actually dual licensed under Eclipse v2 or Apache v2. We can accept other compatible licenses as well.

0xdaryl · 2022-06-21T12:09:57Z

This is not a 0.33 regression. It fails at least as far back as 0.24 with JDK11.

0xdaryl · 2022-06-30T19:41:43Z

A proper fix won't be ready for 0.33 and won't have time to soak in the head stream. This is not a regression, so moving this out to 0.34. @pshipton

connglli · 2022-07-27T12:40:32Z

The following is a much simpler test for this bug (buggy when a() is compiled also at scorching level)

class T {
  void a() {
    int c = 3, d, f = 34829, x = 11801, y = 56517;
    for (d = 4; d < 101; ++d) {
      c += 58095;
      for (int ax$8 = -3901; ax$8 < 2545; ax$8 += 9)
        try {
          short[] ax$4 = {}; ax$4[x] = (short) f;
        } catch (Throwable ax$7) {}
      for (x = 2; x < 52; ++x) { y = (int) 196L; y -= c; }
    }
    System.out.println(y);
  }

  public static void main(String[] args) {
    T t = new T();
    for (int i = 0; i < 10; i++) t.a();
  }
}

hzongaro · 2022-08-25T01:07:41Z

Peter @pshipton, having looked at another problem with ExpressionsSimplification (#15534), I think it might be safer to push this out to the next release rather than trying to rush in a fix for 0.35.

hzongaro · 2022-09-12T20:54:46Z

@connglli, regarding the test you provided in your comment of June 15, it appears that that is a different problem. The incorrect output produced with that test can be avoided by specifying the option -Xjit:disableGlobalVP. It seems to be expose a bug in the Global Value Propagation optimization.

connglli · 2022-09-15T13:04:17Z

Hi, @hzongaro. Thanks for your effort on reviewing my bug reports.

hzongaro · 2022-09-27T22:26:12Z

It looks like the test case in the comment of 27 July exhibits a slightly different problem in ExpressionSimplification than the problem seen in the original test case.

In the original test, Expressions Simplification calculates the value of i11 when it pulls it out of the loop, it multiplies the value of i12 by the wrong number of iterations in calculating the final value of i11.

  void vMeth(long l1, short s2, double d1) {
    int i6, i8, i10, i11 = 12, i12 = 210;
    for (i6 = 7; i6 < 149; ++i6)
      for (i8 = 1; i8 < 4; i8++)
        for (i10 = 1; i10 < 2; ++i10) {
          i11 += i12;
          iFld = i10;
        }
    vMeth_check_sum += i11;
  }

In the case of the smaller test case, the problem is that ExpressionSimplification transforms the assignment in this loop

 for (x = 2; x < 52; ++x) { y = (int) 196L; y -= c; }

into the following, failing to recognize that every iteration of the loop is setting y to 196-c, not performing a summation.

y = 196 - c*50;
for (x = 2; x < 52; ++x);

connglli · 2022-09-28T08:29:32Z

That's interesting. Thanks @hzongaro

hzongaro · 2022-09-28T13:39:30Z

That second problem also affects negation and exclusive-or operators:

class TFor2 {
  void negate(int j) {
    int i = 1;
    for (int x = 2; x < 52; ++x) { i = -j; }
    System.out.println(i);
  }

  void xor(int j) {
    int i = 1;
    for (int x = 2; x < 52; ++x) { i = 2 ^ j; }
    System.out.println(i);
  }

  public static void main(String[] args) {
    TFor2 tfor2 = new TFor2();
    for (int i = 0; i < 10; i++) tfor2.negate(17);
    for (int i = 0; i < 10; i++) tfor2.xor(17);
  }
}

Running with java -Xint TFor2, the output is

Running with java -Xjit:count=1,disableAsyncCompilation,{TFor2.[xn]*}\(optLevel=scorching\) TFor2

-17
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
19
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944
-2090610944

hzongaro · 2023-05-08T12:53:28Z

Hi, @connglli. I wanted to follow up on a comment from last year, asking whether you would be willing to contribute tests that you've created to OpenJ9. You mentioned in a response that you might be able to contribute tests when you had free time. Have you given this any further thought?

connglli · 2023-05-08T17:37:28Z

Sure @hzongaro, I could. I still have some mis-computations/mis-compilations in hand. But firstly, I have to check if they have been fixed by your recent updates. If not, I'll create a PR ASAP after this Friday because I have an important talk on that day and I have to prepare for it these days. This may take a couple of days. Are you or @llxia still willing to help me with the PR as mentioned in the previous comment when I'm all ready?

llxia · 2023-05-08T17:58:02Z

Thanks @connglli . Once you are ready, could you create a new folder under https://github.com/eclipse-openj9/openj9/tree/master/test/functional with your test cases? You wouldn't need to adapt the tests to the testing framework that we use. We will take it from there and incorporate the tests into our framework/pipeline.

You would need to sign the Eclipse Foundation Contributor Agreement. Please see Contributing to Eclipse OpenJ9.

connglli · 2023-05-12T16:05:21Z

@llxia @hzongaro @pshipton I have added some tests in PR 17404. I have prepended a "WIP" in the title as learned from Contributing to Eclipse OpenJ9 such that you can work on it. Feel free to ping me if you need any further help or if I have understood you incorrectly. Thanks!

connglli · 2023-10-11T13:47:41Z

As for the first question, we're working on an automated testing tool based on many existing fuzzers; but it hasn't been released yet. Currently, tests that we have reported is a hybrid of our tool and a fuzzer namely Java*Fuzzer which is licensed by Apache V2. I'm not quite familiar with the licenses, but as far as I know Apache V2 grants you the permission to adapt/modify their code. For ours, you are free to use them. Therefore, I think it's sufficient that you add the Java*Fuzzer's license when including then in your regression test suites.

As for the second question, sorry that currently we don't have that time to make formal contributions using PR following some guidelines. Issuetracker is much more convenient to us at present. But we could consider that when we are free.

A big thanks for your invitation.

@pshipton @hzongaro @llxia Hi all, our work has finally been accepted by SOSP 2023 and our tool has been released under Artemis. For anyone who is interested in the technical details, feel free to navigate to our paper for technical details! Hope Artemis helps you!

pshipton added the userRaised label Jun 13, 2022

pshipton added comp:jit blocker labels Jun 13, 2022

pshipton added this to the Release 0.33 (Java 8, 11, 17, 18) July refresh milestone Jun 13, 2022

connglli mentioned this issue Jun 16, 2022

JDK8 OpenJ9's (Deterministic) Result is Different from That of HotSpot and ART #15347

Open

0xdaryl self-assigned this Jun 16, 2022

This was referenced Jun 16, 2022

JDK8 OpenJ9: Server and Desktop Gives Different Result #15349

Closed

Compilation Fails even from noOpt Level #15369

Closed

0xdaryl modified the milestones: Release 0.33 (Java 8, 11, 17, 18) July refresh, Release 0.34 (Java 19) Jun 30, 2022

connglli mentioned this issue Jul 13, 2022

JDK11 JDK17 Different Result from Xint #15534

Closed

pshipton modified the milestones: Java 19, Release 0.35 (Java 8, 11, 17, 19) Oct refresh Jul 19, 2022

pshipton removed the blocker label Jul 26, 2022

pshipton modified the milestones: Release 0.35 (Java 8, 11, 17, 19) Oct refresh, Release 0.36 (Java 8, 11, 17, 19) Jan refresh Aug 25, 2022

hzongaro mentioned this issue Sep 12, 2022

Incorrect output for test involving switch with fall-through case #15874

Closed

hzongaro mentioned this issue Oct 3, 2022

Fix problems in handling of summation reduction and invariant expressions in Expressions Simplification eclipse-omr/omr#6747

Merged

hzongaro self-assigned this Oct 3, 2022

vijaysun-omr closed this as completed in eclipse-omr/omr#6747 Oct 11, 2022

connglli mentioned this issue Mar 27, 2023

JDK11 OpenJ9 Gives Non-deterministic Results #17033

Open

connglli mentioned this issue May 12, 2023

WIP: Add new tests for OpenJ9 #17404

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK8 OpenJ9 Gives Different Results from HotSpot and Android Runtime #15306

JDK8 OpenJ9 Gives Different Results from HotSpot and Android Runtime #15306

connglli commented Jun 13, 2022

pshipton commented Jun 13, 2022 •

edited

Loading

pshipton commented Jun 13, 2022

pshipton commented Jun 13, 2022 •

edited

Loading

pshipton commented Jun 13, 2022 •

edited

Loading

pshipton commented Jun 13, 2022

0xdaryl commented Jun 14, 2022

connglli commented Jun 15, 2022

pshipton commented Jun 15, 2022

connglli commented Jun 15, 2022

pshipton commented Jun 15, 2022

connglli commented Jun 16, 2022

pshipton commented Jun 16, 2022

0xdaryl commented Jun 21, 2022

0xdaryl commented Jun 30, 2022

connglli commented Jul 27, 2022

hzongaro commented Aug 25, 2022

hzongaro commented Sep 12, 2022

connglli commented Sep 15, 2022

hzongaro commented Sep 27, 2022

connglli commented Sep 28, 2022

hzongaro commented Sep 28, 2022

hzongaro commented May 8, 2023

connglli commented May 8, 2023 •

edited

Loading

llxia commented May 8, 2023

connglli commented May 12, 2023 •

edited

Loading

connglli commented Oct 11, 2023 •

edited

Loading

JDK8 OpenJ9 Gives Different Results from HotSpot and Android Runtime #15306

JDK8 OpenJ9 Gives Different Results from HotSpot and Android Runtime #15306

Comments

connglli commented Jun 13, 2022

Java version

Javac version

Summary of problem

pshipton commented Jun 13, 2022 • edited Loading

pshipton commented Jun 13, 2022

pshipton commented Jun 13, 2022 • edited Loading

pshipton commented Jun 13, 2022 • edited Loading

pshipton commented Jun 13, 2022

0xdaryl commented Jun 14, 2022

connglli commented Jun 15, 2022

pshipton commented Jun 15, 2022

connglli commented Jun 15, 2022

pshipton commented Jun 15, 2022

connglli commented Jun 16, 2022

pshipton commented Jun 16, 2022

0xdaryl commented Jun 21, 2022

0xdaryl commented Jun 30, 2022

connglli commented Jul 27, 2022

hzongaro commented Aug 25, 2022

hzongaro commented Sep 12, 2022

connglli commented Sep 15, 2022

hzongaro commented Sep 27, 2022

connglli commented Sep 28, 2022

hzongaro commented Sep 28, 2022

hzongaro commented May 8, 2023

connglli commented May 8, 2023 • edited Loading

llxia commented May 8, 2023

connglli commented May 12, 2023 • edited Loading

connglli commented Oct 11, 2023 • edited Loading

pshipton commented Jun 13, 2022 •

edited

Loading

pshipton commented Jun 13, 2022 •

edited

Loading

pshipton commented Jun 13, 2022 •

edited

Loading

connglli commented May 8, 2023 •

edited

Loading

connglli commented May 12, 2023 •

edited

Loading

connglli commented Oct 11, 2023 •

edited

Loading