BEF long pause between launch of large set of JUnit tests, and actual execution

[plaird]

An internal workspace has a Java package with 55 test classes, containing 640 test methods. There is a painfully slow pause to launch the JUnit tests for the entire package.

Repro

• Import the package
• Click on the package in the explorer, right click, Run As, JUnit Test
• Eclipse spins for a couple of minutes (this is the issue)
• JUnit tests launch and run as expected
• JUnit view is populated correctly with test results
• Repeat these steps, and the long pause occurs each time

If you click on only one test class, and run all tests within, it is faster but still suffers.

Brief Analysis

Using breakpoints, I was able to find the general cause of the slowness. It seems this is because the launcher is computing a unique classpath per test (which is because Bazel supports that) during launch. We should (I think) just short cut this and rely on the already computed (main+test) classpaths we gave to JDT.

Even if we need to stick with the current scheme (as a short term option), we need to cache the results so subsequent invocations are fast.

Notable code points:

87 targets are analyzed for the run here: BazelJvmClasspath.java

This call to computeUnresolvedRuntimeClasspath() returns 29,000 entries: AbstractJavaLaunchConfigurationDelegate

[plaird]

I did work today to dedupe the test classpath in #388 and #389. This did improve things a bit, but there is more work to be done. There are two phases of time spent when launching tests:

1. computing the aggregate classpath of all tests to be executed
2. invoking the tests in the JUnit test runner, which Eclipse delegates to a separate process

The first phase is well under our control and I think I can continue to make progress there. The first time you run tests in a project it may remain slower than we want, but subsequent executions should become speedy. Sadly, the larger chunk of time is now spent in the second phase.

The long pause you see in between the the "Found binding" messages and the start of the actual tests is in the second phase. What is happening during that long pause? The RemoteTestRunner is indexing classes in all the jars in the classpath. For the internal test case, we are sending 566 jars to the org.eclipse.jdt.internal.junit.runner.RemoteTestRunner. That is about 400 classpath jars, and 150 test class jars (one or two per test class). The project has a large dependency graph so I don't think we can reduce the 400 jars. But we might be able to reduce the number of test class jars. Because the test runner runs in its own process and is an Eclipse owned class, there is a limit to how much we can change the behavior.

To reproduce the problem outside of Eclipse, do this:

1. create a test launcher for all the tests in the project (right click on the project, Run As, JUnit tests)
2. run all the tests once
3. open the Run Configuration dialog, and click on your launcher (under the JUnit section)
4. click on the Show Command Line button, copy the text into an editor
5. remove newlines, and add -junitconsole to the end of the command line
6. paste the command line into a shell, and run it

[plaird]

PR #391 has the details of the major fix that resolved the biggest part of the performance issue. It has to do with "deploy" jars that Bazel generates for some java_test targets. They are unnecessary for BEF, and are very large bloated jars. They were being added to the BEF test classpath, which slowed down test execution dramatically. The PR removed those deploy jars from the test classpath.

[plaird]

This fix will be released in BEF 1.5.1.

A BEF 1.5.1 RC is currently available at this update site: http://opensource.salesforce.com/bazel-eclipse/update-site-release-candidate