Custom java toolchain: Worker process returned an unparseable WorkResponse! (on Bazel 0.16.1)

[davido]

Trying to use custom java_toolchain is failing on latest released Bazel version (0.16.1):

$ bazel version
Build label: 0.16.1

$ bazel build --java_toolchain //tools:error_prone_warnings_toolchain //...
Starting local Bazel server and connecting to it...
ERROR: /home/davido/projects/gerrit2/gitblit/BUILD.bazel:3:1: no such package '@tika//jar': The repository could not be resolved and referenced by '//gitblit:gitblit'
ERROR: Analysis of target '//gitblit:gitblit' failed; build aborted: no such package '@tika//jar': The repository could not be resolved
INFO: Elapsed time: 8.317s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (244 packages loaded)
davido@wizball:~/projects/gerrit2 (master %>)$ rm gitblit
davido@wizball:~/projects/gerrit2 (master %>)$ bazel build --java_toolchain //tools:error_prone_warnings_toolchain //...
ERROR: /home/davido/projects/gerrit2/plugins/metrics-reporter-jmx/BUILD:3:1: no such package '@metrics_jmx//jar': The repository could not be resolved and referenced by '//plugins/metrics-reporter-jmx:metrics-reporter-jmx__plugin'
ERROR: Analysis of target '//plugins/metrics-reporter-jmx:metrics-reporter-jmx__plugin' failed; build aborted: no such package '@metrics_jmx//jar': The repository could not be resolved
INFO: Elapsed time: 6.093s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (162 packages loaded)
davido@wizball:~/projects/gerrit2 (master %>)$ rm plugins/metrics-reporter-jmx
davido@wizball:~/projects/gerrit2 (master %>)$ bazel build --java_toolchain //tools:error_prone_warnings_toolchain :release
INFO: Analysed target //:release (109 packages loaded).
INFO: Found 1 target...
INFO: From Compiling external/com_google_protobuf/src/google/protobuf/descriptor.cc [for host]:
external/com_google_protobuf/src/google/protobuf/descriptor.cc: In member function 'google::protobuf::Symbol google::protobuf::DescriptorPool::NewPlaceholderWithMutexHeld(const string&, google::protobuf::DescriptorPool::PlaceholderType) const':
external/com_google_protobuf/src/google/protobuf/descriptor.cc:3896:58: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::EnumDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     memset(placeholder_enum, 0, sizeof(*placeholder_enum));
                                                          ^
In file included from external/com_google_protobuf/src/google/protobuf/message.h:122,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:28,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.cc:57:
external/com_google_protobuf/src/google/protobuf/descriptor.h:884:26: note: 'class google::protobuf::EnumDescriptor' declared here
 class LIBPROTOBUF_EXPORT EnumDescriptor {
                          ^~~~~~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/descriptor.cc:3910:60: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::EnumValueDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     memset(placeholder_value, 0, sizeof(*placeholder_value));
                                                            ^
In file included from external/com_google_protobuf/src/google/protobuf/message.h:122,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:28,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.cc:57:
external/com_google_protobuf/src/google/protobuf/descriptor.h:1039:26: note: 'class google::protobuf::EnumValueDescriptor' declared here
 class LIBPROTOBUF_EXPORT EnumValueDescriptor {
                          ^~~~~~~~~~~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/descriptor.cc:3929:64: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::Descriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     memset(placeholder_message, 0, sizeof(*placeholder_message));
                                                                ^
In file included from external/com_google_protobuf/src/google/protobuf/message.h:122,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:28,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.cc:57:
external/com_google_protobuf/src/google/protobuf/descriptor.h:225:26: note: 'class google::protobuf::Descriptor' declared here
 class LIBPROTOBUF_EXPORT Descriptor {
                          ^~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/descriptor.cc: In member function 'google::protobuf::FileDescriptor* google::protobuf::DescriptorPool::NewPlaceholderFileWithMutexHeld(const string&) const':
external/com_google_protobuf/src/google/protobuf/descriptor.cc:3963:46: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::FileDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
   memset(placeholder, 0, sizeof(*placeholder));
                                              ^
In file included from external/com_google_protobuf/src/google/protobuf/message.h:122,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:28,
                 from external/com_google_protobuf/src/google/protobuf/descriptor.cc:57:
external/com_google_protobuf/src/google/protobuf/descriptor.h:1282:26: note: 'class google::protobuf::FileDescriptor' declared here
 class LIBPROTOBUF_EXPORT FileDescriptor {
                          ^~~~~~~~~~~~~~
INFO: From Executing genrule @bazel_tools//tools/jdk:platformclasspath [for host]:
/home/davido/.cache/bazel/_bazel_davido/install/8b366f00e926892ca09f727a87c56232/_embedded_binaries/embedded_tools/jdk/jre/lib/rt.jar
ERROR: /home/davido/projects/gerrit2/java/org/eclipse/jgit/BUILD:19:1: Building java/org/eclipse/jgit/libEdit.jar (1 source jar) failed: Worker process returned an unparseable WorkResponse!

Did you try to print something to stdout? Workers aren't allowed to do this, as it breaks the protocol between Bazel and the worker process.

---8<---8<--- Exception details ---8<---8<---
com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
	at com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:115)
	at com.google.protobuf.CodedInputStream$StreamDecoder.skipField(CodedInputStream.java:2100)
	at com.google.protobuf.GeneratedMessageV3.parseUnknownFieldProto3(GeneratedMessageV3.java:303)
	at com.google.devtools.build.lib.worker.WorkerProtocol$WorkResponse.<init>(WorkerProtocol.java:1866)
	at com.google.devtools.build.lib.worker.WorkerProtocol$WorkResponse.<init>(WorkerProtocol.java:1830)
	at com.google.devtools.build.lib.worker.WorkerProtocol$WorkResponse$1.parsePartialFrom(WorkerProtocol.java:2420)
	at com.google.devtools.build.lib.worker.WorkerProtocol$WorkResponse$1.parsePartialFrom(WorkerProtocol.java:2415)
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:221)
	at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:262)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:275)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:280)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
	at com.google.protobuf.GeneratedMessageV3.parseDelimitedWithIOException(GeneratedMessageV3.java:347)
	at com.google.devtools.build.lib.worker.WorkerProtocol$WorkResponse.parseDelimitedFrom(WorkerProtocol.java:2082)
	at com.google.devtools.build.lib.worker.WorkerSpawnRunner.execInWorker(WorkerSpawnRunner.java:324)
	at com.google.devtools.build.lib.worker.WorkerSpawnRunner.actuallyExec(WorkerSpawnRunner.java:154)
	at com.google.devtools.build.lib.worker.WorkerSpawnRunner.exec(WorkerSpawnRunner.java:112)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:95)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
	at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:287)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:294)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:978)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:910)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:120)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:457)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:513)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:227)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:400)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
---8<---8<--- End of exception details ---8<---8<---

---8<---8<--- Start of log ---8<---8<---
-Xbootclasspath/p is no longer a supported option.
ion.
---8<---8<--- End of log ---8<---8<---
Target //:release failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 15.886s, Critical Path: 4.43s
INFO: 629 processes: 575 remote cache hit, 52 linux-sandbox, 1 local, 1 worker.
FAILED: Build did NOT complete successfully

On unreleased 0.17rc1 there is another error:

$ bazel version
Build label: 0.17.0rc1
bazel build --java_toolchain //tools:error_prone_warnings_toolchain //java/com/google/gerrit/server:server
[...]
/bin/bash: line 1:     3 Aborted                 (core dumped) external/bazel_tools/tools/jdk/ijar/ijar bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath-impl.jar bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath.jar
src/main/tools/linux-sandbox-pid1.cc:437: waitpid returned 2
src/main/tools/linux-sandbox-pid1.cc:457: child exited with code 134
src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 134
Target //java/com/google/gerrit/server:server failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.313s, Critical Path: 0.12s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

On Bazel@HEAD (2894b30) it seems to work.

Reproducer: https://gerrit-review.googlesource.com/c/gerrit/+/194045.

[cushon]

In 0.16.1 default_java_toolchain.jvm_opts is defaulting to the JDK 8-compatible jvm flags. You can see that @bazel_tools//tools/jdk:toolchain_java9 overrides them:

bazel/tools/jdk/BUILD

Line 218 in 2da247f

jvm_opts = JDK9_JVM_OPTS,

Overriding them in your custom toolchain will make it work with 0.16.1:

diff --git a/tools/BUILD b/tools/BUILD
index e11db8c64e..890ed0cb50 100644
--- a/tools/BUILD
+++ b/tools/BUILD
@@ -1,4 +1,4 @@
-load("@bazel_tools//tools/jdk:default_java_toolchain.bzl", "default_java_toolchain")
+load("@bazel_tools//tools/jdk:default_java_toolchain.bzl", "default_java_toolchain", "JDK9_JVM_OPTS")
 
 py_binary(
     name = "merge_jars",
@@ -11,6 +11,7 @@ py_binary(
 default_java_toolchain(
     name = "error_prone_warnings_toolchain",
     bootclasspath = ["@bazel_tools//tools/jdk:platformclasspath.jar"],
+    jvm_opts = JDK9_JVM_OPTS,
     package_configuration = [
         ":error_prone",
     ],

That won't be compatible with 0.17, though, since JDK9_JVM_OPTS no longer exists. I guess you could define different toolchains for 0.16.1 and 0.17 as a work-around.

I can't reproduce the problem with 0.17rc1. I run into #5997 (comment) and #5989, but not the error you reported. How do I repro that?

[davido]

I confirm, thar the suggested workaround works as expected on 0.16.1, but that breaks 0.17.rc1, because now I get:

$ bazel build --java_toolchain //tools:error_prone_warnings_toolchain //java/com/google/gerrit/server:server 
ERROR: /home/davido/projects/gerrit2/tools/BUILD:1:1: file '@bazel_tools//tools/jdk:default_java_toolchain.bzl' does not contain symbol 'JDK9_JVM_OPTS'
ERROR: /home/davido/projects/gerrit2/tools/BUILD:18:16: Traceback (most recent call last):
	File "/home/davido/projects/gerrit2/tools/BUILD", line 15
		default_java_toolchain(name = "error_prone_warnings_too...", <4 more arguments>)
	File "/home/davido/projects/gerrit2/tools/BUILD", line 18, in default_java_toolchain
		JDK9_JVM_OPTS
name 'JDK9_JVM_OPTS' is not defined

Is there a way to import things conditionally?

Reproducer: https://gerrit-review.googlesource.com/c/gerrit/+/194045

[cushon]

That won't be compatible with 0.17, though, since JDK9_JVM_OPTS no longer exists.

I confirm, thar the suggested workaround works as expected on 0.16.1, but that breaks 0.17.rc1

Right. I'm not aware of a conditional import mechanism. You could define different toolchains for 0.16 and 0.17, or inline those JVM flags instead of using the constant.

Is the core dump you mentioned seeing with 0.17rc1 in your original comment reproducible, or was that a transient failure?

[davido]

I can't reproduce the problem with 0.17rc1.

[...]

Is the core dump you mentioned seeing with 0.17rc1 in your original comment reproducible, or was that a transient failure?

Never mind, I re-downloaded 0.17rc1 and confirmed it works with restrictions, that you've mentioned. Apparently I had the old 0.17rc1 download, that was re-cut recently. Sorry that I messed that up.

[jowagner]

An alternative workaround should then be to respect https://docs.bazel.build/versions/master/install-compile-source.html and use JDK 8. I can confirm that this works: I just built 0.16.1 successfully on openSUSE Leap 15.0 with JDK 8.

(I uninstalled JDKs 10 and 11 as openSUSE doesn't have update-java-alternatives and changing a handful of alternatives with update-alternatives seems too much hassle.)