Bootstrapped bazel does not support bzlmod lock file #20501

avdv · 2023-12-12T14:05:08Z

Description of the bug:

When bootstrapping bazel from scratch using a distribution zip archive, the resulting bazel binary's A-server.jar does not contain the *_GsonTypeAdapter.class files which are needed for reading / writing the MODULE.bazel.lock file.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

download https://github.com/bazelbuild/bazel/releases/download/6.4.0/bazel-6.4.0-dist.zip
mkdir build && cd build
unzip ~/Downloads/bazel-6.4.0-dist.zip
env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
unzip output/bazel A-server.jar
unzip -l A-server.jar | grep -F _GsonTypeAdapter.class (this should produce some output, but does not)

In a project with bzlmod enabled, trying to update the MODULE.bazel.lock file:

$ output/bazel mod deps --lockfile_mode=update 
...
$ cat MODULE.bazel.lock
{}

When trying to run inside a project with a valid lock file, bazel crashes:

$ output/bazel mod deps --lockfile_mode=update
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue$$Lambda$223/0x00000008401fe440@e6399be' (requested by nodes 'com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphValue$$Lambda$388/0x0000000840452440@260c34d0')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:633)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:365)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: Failed to invoke public com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue() with no args
	at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:113)
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:212)
	at com.google.gson.Gson.fromJson(Gson.java:932)
	at com.google.gson.Gson.fromJson(Gson.java:897)
	at com.google.gson.Gson.fromJson(Gson.java:846)
	at com.google.gson.Gson.fromJson(Gson.java:817)
	at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.getLockfileValue(BazelLockFileFunction.java:112)
	at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.compute(BazelLockFileFunction.java:87)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:562)
	... 7 more
Caused by: java.lang.InstantiationException
	at java.base/jdk.internal.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:110)
	... 15 more

Which operating system are you running Bazel on?

Ubuntu Linux

What is the output of `bazel info release`?

n/a

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

Previously: #19672

Any other information, logs, or outputs that you want to share?

Contrary to when bootstrapping from scratch, building bazel with bazel works:

$ BAZEL=/usr/bin/bazel ./compile.sh 
🍃  Building Bazel with Bazel.
.2023/12/12 13:06:16 Downloading https://releases.bazel.build/6.4.0/release/bazel-6.4.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
...

Target //src:bazel_nojdk up-to-date:
  bazel-bin/src/bazel_nojdk
INFO: Elapsed time: 431.183s, Critical Path: 138.05s
INFO: 2947 processes: 63 internal, 2084 local, 800 worker.
INFO: Build completed successfully, 2947 total actions

Build successful! Binary is here: /home/build/testme/output/bazel

$ unzip output/bazel A-server.jar
Archive:  output/bazel
  inflating: A-server.jar
$ unzip -l A-server.jar | grep -F _GsonTypeAdapter.class
     3981  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/AttributeValues_GsonTypeAdapter.class
     7945  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/BazelLockFileValue_GsonTypeAdapter.class
     6648  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/BzlmodFlagsAndEnvVars_GsonTypeAdapter.class
     4605  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/GsonTypeAdapterUtil_RootModuleFileEscapingLocation_GsonTypeAdapter.class
    10559  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/InterimModule_GsonTypeAdapter.class
     6948  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/LockFileModuleExtension_GsonTypeAdapter.class
     3867  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/ModuleExtensionEvalFactors_GsonTypeAdapter.class
     5393  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/ModuleExtensionMetadata_GsonTypeAdapter.class
     9548  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/ModuleExtensionUsage_GsonTypeAdapter.class
     8541  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/Module_GsonTypeAdapter.class
     4684  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/RepoSpec_GsonTypeAdapter.class
     5475  01-01-1980 00:00   com/google/devtools/build/lib/bazel/bzlmod/Tag_GsonTypeAdapter.class

The text was updated successfully, but these errors were encountered:

Wyverald · 2023-12-14T21:35:47Z

@meteorcloudy any ideas?

meteorcloudy · 2023-12-15T14:17:03Z

I'll take a look at this

meteorcloudy · 2023-12-15T15:51:24Z

/cc @SalmaSamy since this is related to lockfile

meteorcloudy · 2023-12-15T16:14:10Z

From #20535

The missing classes are generated by an annotation processor, which may not be run during the bootstrap build.

@hvadehra I need some help from a Java expert. Initially I thought this is a problem in the bootstrap java toolchain

But simply adding the relevant jars didn't fix the issue.

diff --git a/third_party/BUILD b/third_party/BUILD
index 912beec0fe..35210f3d7b 100644
--- a/third_party/BUILD
+++ b/third_party/BUILD
@@ -251,6 +251,11 @@ distrib_jar_filegroup(
     srcs = [
         "@maven//:com_google_auto_value_auto_value_annotations_file",
         "@maven//:com_google_auto_value_auto_value_file",
+        "@maven//:com_ryanharter_auto_value_auto_value_gson_runtime_file",
+        "@maven//:com_ryanharter_auto_value_auto_value_gson_extension_file",
+        "@maven//:com_ryanharter_auto_value_auto_value_gson_factory_file",
+        "@maven//:com_squareup_javapoet_file",
+        "@maven//:com_google_code_gson_gson_file",
     ],
     enable_distributions = ["debian"],
 )

And from the issue descriptions, building with an existing bazel binary with the same bootstrap toolchain resolves the problem. So something is missing from the raw bootstrap bazel we compiled from javac? How does annotation processor work?

meteorcloudy · 2023-12-18T15:45:58Z

Found a fix in #20587, somehow the target/source java version matters.

iancha1992 · 2023-12-18T19:00:23Z

@bazel-io fork 7.0.1

For the bootstrap VanillaJavaBuilder, it is important that the AutoValue plugin classes do not end up in the deploy jar. If they do, the processor class is loaded from the app classloader (instead of the processor path classloader). When this happens, AutoValueProcessor uses the app classloader to load extensions (such as auto-value-gson needed by bzlmod), instead of the processor path classloader. Unless all extensions are also correctly present in the app classloder (i.e. in the VanillaJavaBuilder deploy jar), they won't be loaded. Unfortunately, simply adding the jars to the deploy jar is insufficient, we need to also correctly merge the `META-INF/service/...` files as well, which does not happen in our bootstrap java_binary/java_library rules. So, to fix, we remove the auto value plugins from the deploy jar, and use them only to compile our sources that require them. Added a regression test for the crash. We should probably improve it so that we can actually build with the bootstapped Bazel in tests without network access. But I don't think we need block this patch release on that. Fixes bazelbuild#20501 PiperOrigin-RevId: 592266992 Change-Id: I08ac91ee74140df0c4f22ad2553a8c017b497181

For the bootstrap VanillaJavaBuilder, it is important that the AutoValue plugin classes do not end up in the deploy jar. If they do, the processor class is loaded from the app classloader (instead of the processor path classloader). When this happens, AutoValueProcessor uses the app classloader to load extensions (such as auto-value-gson needed by bzlmod), instead of the processor path classloader. Unless all extensions are also correctly present in the app classloder (i.e. in the VanillaJavaBuilder deploy jar), they won't be loaded. Unfortunately, simply adding the jars to the deploy jar is insufficient, we need to also correctly merge the `META-INF/service/...` files as well, which does not happen in our bootstrap java_binary/java_library rules. So, to fix, we remove the auto value plugins from the deploy jar, and use them only to compile our sources that require them. Added a regression test for the crash. We should probably improve it so that we can actually build with the bootstapped Bazel in tests without network access. But I don't think we need block this patch release on that. Fixes #20501 Commit 2a2def8 PiperOrigin-RevId: 592266992 Change-Id: I08ac91ee74140df0c4f22ad2553a8c017b497181 Co-authored-by: Googler <[email protected]>

This works around an issue with bootstrapping bazel from scratch, see bazelbuild/bazel#20501 Fixes NixOS#273500

avdv · 2023-12-20T09:17:30Z

Will this fix be backported to 6.x ?

iancha1992 · 2023-12-20T18:44:40Z

@bazel-io fork 6.5.0

iancha1992 · 2024-01-05T23:52:58Z

@bazel-io fork 7.1.0

For the bootstrap VanillaJavaBuilder, it is important that the AutoValue plugin classes do not end up in the deploy jar. If they do, the processor class is loaded from the app classloader (instead of the processor path classloader). When this happens, AutoValueProcessor uses the app classloader to load extensions (such as auto-value-gson needed by bzlmod), instead of the processor path classloader. Unless all extensions are also correctly present in the app classloder (i.e. in the VanillaJavaBuilder deploy jar), they won't be loaded. Unfortunately, simply adding the jars to the deploy jar is insufficient, we need to also correctly merge the `META-INF/service/...` files as well, which does not happen in our bootstrap java_binary/java_library rules. So, to fix, we remove the auto value plugins from the deploy jar, and use them only to compile our sources that require them. Added a regression test for the crash. We should probably improve it so that we can actually build with the bootstapped Bazel in tests without network access. But I don't think we need block this patch release on that. Fixes bazelbuild#20501 PiperOrigin-RevId: 592266992 Change-Id: I08ac91ee74140df0c4f22ad2553a8c017b497181

For the bootstrap VanillaJavaBuilder, it is important that the AutoValue plugin classes do not end up in the deploy jar. If they do, the processor class is loaded from the app classloader (instead of the processor path classloader). When this happens, AutoValueProcessor uses the app classloader to load extensions (such as auto-value-gson needed by bzlmod), instead of the processor path classloader. Unless all extensions are also correctly present in the app classloder (i.e. in the VanillaJavaBuilder deploy jar), they won't be loaded. Unfortunately, simply adding the jars to the deploy jar is insufficient, we need to also correctly merge the `META-INF/service/...` files as well, which does not happen in our bootstrap java_binary/java_library rules. So, to fix, we remove the auto value plugins from the deploy jar, and use them only to compile our sources that require them. Added a regression test for the crash. We should probably improve it so that we can actually build with the bootstapped Bazel in tests without network access. But I don't think we need block this patch release on that. Fixes #20501 PiperOrigin-RevId: 592266992 Change-Id: I08ac91ee74140df0c4f22ad2553a8c017b497181 Co-authored-by: Googler <[email protected]>

iancha1992 · 2024-01-17T00:27:06Z

A fix for this issue has been included in Bazel 6.5.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks!

avdv added type: bug untriaged labels Dec 12, 2023

avdv assigned iancha1992, rakshitasingh05 and sgowroji Dec 12, 2023

This was referenced Dec 12, 2023

Bazel creates an empty lockfile and crashes in subsequent invocation #19672

Closed

bazel_6: Fix empty bzlmod lock file NixOS/nixpkgs#273774

Closed

sgowroji added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. area-Bzlmod Bzlmod-specific PRs, issues, and feature requests labels Dec 12, 2023

sgowroji unassigned sgowroji, iancha1992 and rakshitasingh05 Dec 12, 2023

fmeum mentioned this issue Dec 14, 2023

bazel 7.0.0 bootstrap fails: bazel 7.0.0 bootstrap fails #20535

Closed

meteorcloudy self-assigned this Dec 15, 2023

meteorcloudy added P1 I'll work on this now. (Assignee required) and removed untriaged labels Dec 15, 2023

meteorcloudy mentioned this issue Dec 18, 2023

Fix bootstrapped Bazel binary #20587

Closed

bazel-io mentioned this issue Dec 18, 2023

[7.0.1] Bootstrapped bazel does not support bzlmod lock file #20593

Closed

copybara-service bot closed this as completed in 2a2def8 Dec 19, 2023

bazel-io mentioned this issue Dec 19, 2023

[7.0.1] Fix bootstrapped Bazel binary #20612

Merged

avdv added a commit to avdv/nixpkgs that referenced this issue Dec 20, 2023

bazel_6: Fix bootstrapping bazel with bzlmod lock file support

b75fc95

This works around an issue with bootstrapping bazel from scratch, see bazelbuild/bazel#20501 Fixes NixOS#273500

bazel-io mentioned this issue Dec 20, 2023

[6.5.0] Bootstrapped bazel does not support bzlmod lock file #20624

Closed

bazel-io mentioned this issue Jan 5, 2024

[7.1.0] Bootstrapped bazel does not support bzlmod lock file #20764

Closed

meteorcloudy mentioned this issue Jan 9, 2024

[6.5.0] Fix bootstrapped Bazel binary #20804

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrapped bazel does not support bzlmod lock file #20501

Bootstrapped bazel does not support bzlmod lock file #20501

avdv commented Dec 12, 2023 •

edited

Loading

Wyverald commented Dec 14, 2023

meteorcloudy commented Dec 15, 2023

meteorcloudy commented Dec 15, 2023

meteorcloudy commented Dec 15, 2023 •

edited

Loading

meteorcloudy commented Dec 18, 2023 •

edited

Loading

iancha1992 commented Dec 18, 2023

avdv commented Dec 20, 2023

iancha1992 commented Dec 20, 2023

iancha1992 commented Jan 5, 2024

iancha1992 commented Jan 17, 2024

Bootstrapped bazel does not support bzlmod lock file #20501

Bootstrapped bazel does not support bzlmod lock file #20501

Comments

avdv commented Dec 12, 2023 • edited Loading

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

Wyverald commented Dec 14, 2023

meteorcloudy commented Dec 15, 2023

meteorcloudy commented Dec 15, 2023

meteorcloudy commented Dec 15, 2023 • edited Loading

meteorcloudy commented Dec 18, 2023 • edited Loading

iancha1992 commented Dec 18, 2023

avdv commented Dec 20, 2023

iancha1992 commented Dec 20, 2023

iancha1992 commented Jan 5, 2024

iancha1992 commented Jan 17, 2024

avdv commented Dec 12, 2023 •

edited

Loading

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

meteorcloudy commented Dec 15, 2023 •

edited

Loading

meteorcloudy commented Dec 18, 2023 •

edited

Loading