Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel's java tests sometimes fail with all test cases passed but TEST_PREMATURE_EXIT_FILE exists #24655

Closed
meteorcloudy opened this issue Dec 12, 2024 · 2 comments
Assignees
Labels
P1 I'll work on this now. (Assignee required) team-Rules-Java Issues for Java rules type: bug untriaged

Comments

@meteorcloudy
Copy link
Member

Description of the bug:

Example: https://buildkite.com/bazel/bazel-bazel/builds/30505#0193b7e6-4a47-41a7-b5f8-cf4ffa391104

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

So far only seen on CI, not reproducible locally

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

8.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

Yes, only appeared with Bazel 8.0.0, probably triggered by #23284, but it may not be the root cause.

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

@meteorcloudy meteorcloudy added team-Rules-Java Issues for Java rules P1 I'll work on this now. (Assignee required) labels Dec 12, 2024
@meteorcloudy
Copy link
Member Author

@bazel-io fork 8.0.1

@meteorcloudy
Copy link
Member Author

meteorcloudy commented Dec 12, 2024

Found a way to reproduce locally:

  1. Make a simple java test (//examples/java-native/src/test/java/com/example/myproject:hello) flaky by sleep 5~10s randomly and --test_timeout=7
$ git diff examples/java-native/src/test/java/com/example/myproject/TestHello.java
diff --git a/examples/java-native/src/test/java/com/example/myproject/TestHello.java b/examples/java-native/src/test/java/com/example/myproject/TestHello.java
index e8b775847f..5e0328a63b 100644
--- a/examples/java-native/src/test/java/com/example/myproject/TestHello.java
+++ b/examples/java-native/src/test/java/com/example/myproject/TestHello.java
@@ -1,5 +1,7 @@
 package com.example.myproject;

+import java.util.Random;
+
 import static org.junit.Assert.assertEquals;

 import org.junit.Test;
@@ -15,12 +17,15 @@ import java.nio.charset.StandardCharsets;
  * args, {@link Greeter} should print "Hello &lt;arg[0]&gt;".</p>
  */
 public class TestHello {
+  Random random = new Random();

   @Test
   public void testNoArgument() throws Exception {
     ByteArrayOutputStream out = new ByteArrayOutputStream();
     Greeter.out = new PrintStream(out);
     Greeter.main();
+    int sleepTime = 5000 + random.nextInt(5000);
+    Thread.sleep(sleepTime);
     assertEquals("Hello world", new String(out.toByteArray(), StandardCharsets.UTF_8).trim());
   }

  1. Run with disk cache and flaky test attempt flags:
bazel clean && rm -rf /tmp/cache_1 && bazel test examples/java-native/src/test/java/com/example/myproject:hello --test_timeout=7 --disk_cache=/tmp/cache_1 --flaky_test_attempts=3
  1. Run the test again
bazel test examples/java-native/src/test/java/com/example/myproject:hello --test_timeout=7 --disk_cache=/tmp/cache_1 --flaky_test_attempts=3

If step 2 fails, then step 3 should be able to reproduce the failure with a given chance. And once it's reproduced and cached, rerunning will show the same error again.

bazel-io pushed a commit to bazel-io/bazel that referenced this issue Dec 12, 2024
`TEST_PREMATURE_EXIT_FILE` was not cleaned up between test attempts (--flaky_test_attempts).

Fixes bazelbuild#24655

Closes bazelbuild#24657.

PiperOrigin-RevId: 705495967
Change-Id: Ic4d4be1e72230c3a6bc45c9f372d1188fefad201
github-merge-queue bot pushed a commit that referenced this issue Dec 12, 2024
`TEST_PREMATURE_EXIT_FILE` was not cleaned up between test attempts
(--flaky_test_attempts).

Fixes #24655

Closes #24657.

PiperOrigin-RevId: 705495967
Change-Id: Ic4d4be1e72230c3a6bc45c9f372d1188fefad201

Commit
b33d51f

Co-authored-by: Yun Peng <[email protected]>
copybara-service bot pushed a commit that referenced this issue Dec 13, 2024
Partially rollback 51afbd6 due to #24655, which is causing flaky test failures on CI.

We'll upgrade to Bazel 8.0.1 as soon as it's out.

PiperOrigin-RevId: 705822532
Change-Id: Ia38381d1820d5e41f0b1d2086d712904b5ea10fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) team-Rules-Java Issues for Java rules type: bug untriaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants