Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.0.0] Regression in 7.0.0rc4: singlejar doesn't follow symbolic link (after flip of --incompatible_sandbox_hermetic_tmp option per default) #20262

Closed
davido opened this issue Nov 18, 2023 · 11 comments
Labels
team-Local-Exec Issues and PRs for the Execution (Local) team type: bug untriaged

Comments

@davido
Copy link
Contributor

davido commented Nov 18, 2023

Description of the bug:

After upgrading Bazel from 7.0.0rc3 to 7.0.0rc4 trying to build Gerrit plugin, that is linked to the Gerrit workspace per symbolic link, stopped working:

$ bazeldev build plugins/oauth
INFO: Invocation ID: ba7baca5-1122-4269-958d-66e8a93f1af0
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=158
INFO: Reading rc options for 'build' from /home/davido/projects/gerrit2/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod
INFO: Reading rc options for 'build' from /home/davido/projects/gerrit2/.bazelrc:
  'build' options: --workspace_status_command=python3 ./tools/workspace_status.py --repository_cache=~/.gerritcodereview/bazel-cache/repository --action_env=PATH --disk_cache=~/.gerritcodereview/bazel-cache/cas --java_language_version=21 --java_runtime_version=remotejdk_21 --tool_java_language_version=21 --tool_java_runtime_version=remotejdk_21 --incompatible_strict_action_env --announce_rc
INFO: Analyzed target //plugins/oauth:oauth (221 packages loaded, 5381 targets configured).
ERROR: /home/davido/projects/gerrit2/plugins/oauth/BUILD:10:14: Building Java resource jar failed: (Exit 1): singlejar_local failed: error executing JavaResourceJar command (from target //plugins/oauth:oauth__plugin) external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local --normalize --dont_change_compression --exclude_build_data --output ... (remaining 15 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
singlejar_local: ./src/tools/singlejar/mapped_file_posix.inc:42: open plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:927: stat plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:961: plugins/oauth/src/main/resources/Documentation/build.md: No such file or directory
Target //plugins/oauth:oauth failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 14.366s, Critical Path: 9.00s
INFO: 15 processes: 6 internal, 7 linux-sandbox, 2 worker.
ERROR: Build did NOT complete successfully
davido@localhost:~/projects/gerrit2 (master *%>)$ bazelisk build -s --verbose_failures plugins/oauth
INFO: Invocation ID: e4e8f570-ac4b-4de9-b658-caf5498bf0b7
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=158
INFO: Reading rc options for 'build' from /home/davido/projects/gerrit2/.bazelrc:
  Inherited 'common' options: --noenable_bzlmod
INFO: Reading rc options for 'build' from /home/davido/projects/gerrit2/.bazelrc:
  'build' options: --workspace_status_command=python3 ./tools/workspace_status.py --repository_cache=~/.gerritcodereview/bazel-cache/repository --action_env=PATH --disk_cache=~/.gerritcodereview/bazel-cache/cas --java_language_version=21 --java_runtime_version=remotejdk_21 --tool_java_language_version=21 --tool_java_runtime_version=remotejdk_21 --incompatible_strict_action_env --announce_rc
INFO: Analyzed target //plugins/oauth:oauth (0 packages loaded, 0 targets configured).
SUBCOMMAND: # //plugins/oauth:oauth__plugin [action 'Building Java resource jar', configuration: 4dfb2762efb012ebd8754cb2aa2b486f128ed68c9ebd530cc4bb378c1d1abadf, execution platform: @@local_config_platform//:host, mnemonic: JavaResourceJar]
(cd /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/execroot/gerrit && \
  exec env - \
    PATH=/home/davido/.cache/bazelisk/downloads/bazelbuild/bazel-7.0.0rc4-linux-x86_64/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/google-cloud-sdk/bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/bin:/usr/local/bin:/usr/bin:/bin \
  external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local --normalize --dont_change_compression --exclude_build_data --output bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin.jar --sources bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin-class.jar --resources plugins/oauth/src/main/resources/Documentation/build.md:Documentation/build.md plugins/oauth/src/main/resources/Documentation/config.md:Documentation/config.md plugins/oauth/src/main/resources/Documentation/images/github-1.png:Documentation/images/github-1.png plugins/oauth/src/main/resources/Documentation/images/github-2.png:Documentation/images/github-2.png plugins/oauth/src/main/resources/Documentation/images/gitlab-1.png:Documentation/images/gitlab-1.png plugins/oauth/src/main/resources/Documentation/images/gitlab-2.png:Documentation/images/gitlab-2.png plugins/oauth/src/main/resources/Documentation/images/google-1.png:Documentation/images/google-1.png plugins/oauth/src/main/resources/Documentation/images/google-2.png:Documentation/images/google-2.png plugins/oauth/src/main/resources/Documentation/images/google-3.png:Documentation/images/google-3.png plugins/oauth/src/main/resources/Documentation/images/google-4.png:Documentation/images/google-4.png plugins/oauth/src/main/resources/Documentation/images/google-5.png:Documentation/images/google-5.png)
# Configuration: 4dfb2762efb012ebd8754cb2aa2b486f128ed68c9ebd530cc4bb378c1d1abadf
# Execution platform: @@local_config_platform//:host
ERROR: /home/davido/projects/gerrit2/plugins/oauth/BUILD:10:14: Building Java resource jar failed: (Exit 1): singlejar_local failed: error executing JavaResourceJar command (from target //plugins/oauth:oauth__plugin) 
  (cd /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/87/execroot/gerrit && \
  exec env - \
    PATH=/home/davido/.cache/bazelisk/downloads/bazelbuild/bazel-7.0.0rc4-linux-x86_64/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/google-cloud-sdk/bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/bin:/usr/local/bin:/usr/bin:/bin \
  external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local --normalize --dont_change_compression --exclude_build_data --output bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin.jar --sources bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin-class.jar --resources plugins/oauth/src/main/resources/Documentation/build.md:Documentation/build.md plugins/oauth/src/main/resources/Documentation/config.md:Documentation/config.md plugins/oauth/src/main/resources/Documentation/images/github-1.png:Documentation/images/github-1.png plugins/oauth/src/main/resources/Documentation/images/github-2.png:Documentation/images/github-2.png plugins/oauth/src/main/resources/Documentation/images/gitlab-1.png:Documentation/images/gitlab-1.png plugins/oauth/src/main/resources/Documentation/images/gitlab-2.png:Documentation/images/gitlab-2.png plugins/oauth/src/main/resources/Documentation/images/google-1.png:Documentation/images/google-1.png plugins/oauth/src/main/resources/Documentation/images/google-2.png:Documentation/images/google-2.png plugins/oauth/src/main/resources/Documentation/images/google-3.png:Documentation/images/google-3.png plugins/oauth/src/main/resources/Documentation/images/google-4.png:Documentation/images/google-4.png plugins/oauth/src/main/resources/Documentation/images/google-5.png:Documentation/images/google-5.png)
# Configuration: 4dfb2762efb012ebd8754cb2aa2b486f128ed68c9ebd530cc4bb378c1d1abadf
# Execution platform: @@local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
singlejar_local: ./src/tools/singlejar/mapped_file_posix.inc:42: open plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:927: stat plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:961: plugins/oauth/src/main/resources/Documentation/build.md: No such file or directory
Target //plugins/oauth:oauth failed to build
INFO: Elapsed time: 1.309s, Critical Path: 1.10s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

With the --sandbox_debug, this additional output is issued:

DEBUG: Sandbox debug output for JavaResourceJar //plugins/oauth:oauth__plugin: 1700367499.080166351: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1700367499.080213264: src/main/tools/linux-sandbox.cc:165: Netns is 0
1700367499.080218180: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1700367499.080505680: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 9186
1700367499.080543817: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1700367499.080669854: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1700367499.080807803: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/execroot -> /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-execroot
1700367499.080832490: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/execroot -> /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-working-directory
1700367499.080852905: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remote_java_tools_linux -> /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/0
1700367499.080864331: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/davido/projects/gerrit2 -> /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/1
1700367499.080875738: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp -> /tmp
1700367499.080888857: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp/bazel-execroot/gerrit
1700367499.080897508: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1700367499.080911442: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1700367499.080920426: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /tmp/bazel-working-directory/gerrit
1700367499.081008406: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1700367499.081021132: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1700367499.081029895: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1700367499.081037629: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1700367499.081045515: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1700367499.081053327: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1700367499.081060474: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1700367499.081067983: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1700367499.081079253: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1700367499.081088505: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1700367499.081096986: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1700367499.081105632: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1700367499.081113577: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug/tracing
1700367499.081120291: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /sys/kernel/debug/tracing, nullptr, 2101281, nullptr) failure (Permission denied) ignored
1700367499.081136169: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1700367499.081144889: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1700367499.081154432: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1700367499.081186913: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1700367499.081195684: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1700367499.081208379: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1700367499.081225629: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1700367499.081234923: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1700367499.081243334: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1700367499.081251525: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/0
1700367499.081259273: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1700367499.081266797: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-execroot
1700367499.081294971: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-working-directory
1700367499.081303978: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/0
1700367499.081312064: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/1
1700367499.081319655: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1700367499.081326170: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1700367499.081333149: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1700367499.081340108: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1700367499.081346965: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1700367499.081353822: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-execroot/gerrit
1700367499.081361493: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1700367499.081368139: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1700367499.081374774: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-execroot/gerrit
1700367499.081381513: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1700367499.081388025: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1700367499.081394786: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1700367499.081405133: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1700367499.081413077: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-working-directory/gerrit
1700367499.081458190: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1700367499.081640144: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1700367499.085365495: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x100
1700367499.085380358: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 1
1700367499.085810222: src/main/tools/linux-sandbox.cc:243: child exited normally with code 1
ERROR: /home/davido/projects/gerrit2/plugins/oauth/BUILD:10:14: Building Java resource jar failed: (Exit 1): linux-sandbox failed: error executing JavaResourceJar command 
  (cd /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/execroot/gerrit && \
  exec env - \
    PATH=/home/davido/.cache/bazelisk/downloads/bazelbuild/bazel-7.0.0rc4-linux-x86_64/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/google-cloud-sdk/bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/bin:/usr/local/bin:/usr/bin:/bin \
    TMPDIR=/tmp \
  /home/davido/.cache/bazel/_bazel_davido/install/abc3756db1b1aa394c5545504b855499/linux-sandbox -W /tmp/bazel-working-directory/gerrit -t 15 -w /tmp/bazel-execroot/gerrit -w /tmp -w /dev/shm -M /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/execroot -m /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-execroot -M /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/execroot -m /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-working-directory -M /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remote_java_tools_linux -m /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/0 -M /home/davido/projects/gerrit2 -m /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp/bazel-source-roots/1 -M /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/_hermetic_tmp -m /tmp -S /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/stats.out -D /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox/linux-sandbox/155/debug.out -- external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local --normalize --dont_change_compression --exclude_build_data --output bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin.jar --sources bazel-out/k8-fastbuild/bin/plugins/oauth/liboauth__plugin-class.jar --resources plugins/oauth/src/main/resources/Documentation/build.md:Documentation/build.md plugins/oauth/src/main/resources/Documentation/config.md:Documentation/config.md plugins/oauth/src/main/resources/Documentation/images/github-1.png:Documentation/images/github-1.png plugins/oauth/src/main/resources/Documentation/images/github-2.png:Documentation/images/github-2.png plugins/oauth/src/main/resources/Documentation/images/gitlab-1.png:Documentation/images/gitlab-1.png plugins/oauth/src/main/resources/Documentation/images/gitlab-2.png:Documentation/images/gitlab-2.png plugins/oauth/src/main/resources/Documentation/images/google-1.png:Documentation/images/google-1.png plugins/oauth/src/main/resources/Documentation/images/google-2.png:Documentation/images/google-2.png plugins/oauth/src/main/resources/Documentation/images/google-3.png:Documentation/images/google-3.png plugins/oauth/src/main/resources/Documentation/images/google-4.png:Documentation/images/google-4.png plugins/oauth/src/main/resources/Documentation/images/google-5.png:Documentation/images/google-5.png)
singlejar_local: ./src/tools/singlejar/mapped_file_posix.inc:42: open plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:927: stat plugins/oauth/src/main/resources/Documentation/build.md:: No such file or directory
singlejar_local: src/tools/singlejar/output_jar.cc:961: plugins/oauth/src/main/resources/Documentation/build.md: No such file or directory
Target //plugins/oauth:oauth failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.346s, Critical Path: 1.07s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

In bazel 7.0.0rc3 it works as expected.

In the above example, the actually location of the oauth-plugin:

davido@localhost:~/projects/gerrit2 (master *%>)$ ls -all plugins/oauth
lrwxrwxrwx 1 davido users 11 Nov 19 00:20 plugins/oauth -> ../../oauth

Which category does this issue belong to?

Java Rules

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clone gerrit recursively say in $HOME/projects/gerrit.

Clone gerrit-oauth plugin say in $HOME/projects/oauth.

Create symbolic link in gerrit's plugin directory:

$ cd $HOME/projects/gerrit/plugins
$ ln -s ../../oauth .

Copy external_plugin_deps.bzl from oauth Workspace in gerrit/plugins directory:

$ cd $HOME/projects/gerrit
$ cp ../oauth/external_plugin_deps.bzl plugins/external_plugin_deps.bzl

Try to build the plugin with 7.0.0rc4, using this CL: [1]
Important, make sure, that the parent change is also included: [2], to unflip --enable-bzlmod bit.

Build gerrit-oauth plugin in gerrit workspace mode like this:

$ cd $HOME/projects/gerrit
$ bazeldev build plugins/oauth

[1] https://gerrit-review.googlesource.com/c/gerrit/+/387360
[2] https://gerrit-review.googlesource.com/c/gerrit/+/393394

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

7.0.0.rc4

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Yes, it's a regression, it works as expected in Bazel 7.0.0rc3.

It seems to be related to --incompatible_sandbox_hermetic_tmp option flip per default in:
#19943

Have you found anything relevant by searching the web?

This issue seems to be related.

Any other information, logs, or outputs that you want to share?

Related build rule in oauth-plugin is:

gerrit_plugin(
    name = "oauth",
    srcs = glob(["src/main/java/**/*.java"]),
    manifest_entries = [
        "Gerrit-PluginName: gerrit-oauth-provider",
        "Gerrit-Module: com.googlesource.gerrit.plugins.oauth.Module",
        "Gerrit-HttpModule: com.googlesource.gerrit.plugins.oauth.HttpModule",
        "Gerrit-InitStep: com.googlesource.gerrit.plugins.oauth.InitOAuth",
        "Implementation-Title: Gerrit OAuth authentication provider",
        "Implementation-URL: https://github.com/davido/gerrit-oauth-provider",
    ],
    resources = glob(["src/main/resources/**/*"]),
    deps = [
        "@commons-codec//jar:neverlink",
        "@jackson-core//jar",
        "@jackson-databind//jar",
        "@scribejava-apis//jar",
        "@scribejava-core//jar",
    ],
)

This rule is expanded to:

$ bazeldev query --output=build plugins/oauth:oauth__plugin
# /home/davido/projects/gerrit2/plugins/oauth/BUILD:10:14
java_library(
  name = "oauth__plugin",
  visibility = ["//visibility:public"],
  tags = ["__JAVA_RULES_MIGRATION_DO_NOT_USE_WILL_BREAK__"],
  generator_name = "oauth",
  generator_function = "gerrit_plugin",
  generator_location = "plugins/oauth/BUILD:10:14",
  srcs = ["//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/AirVantageApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/AirVantageOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/Auth0Api.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/Auth0OAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/AuthentikApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/AuthentikOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/AzureActiveDirectoryService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/BitbucketApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/BitbucketOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/CasApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/CasOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/DexApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/DexOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/DisabledOAuthLoginProvider.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/Facebook2Api.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/FacebookOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/GitHub2Api.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/GitHubOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/GitLabApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/GitLabOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/Google2Api.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/GoogleOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/HttpModule.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/InitOAuth.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/KeycloakApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/KeycloakOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/LemonLDAPApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/LemonLDAPOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/Module.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/PhabricatorApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/PhabricatorOAuthService.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/TuleapApi.java", "//plugins/oauth:src/main/java/com/googlesource/gerrit/plugins/oauth/TuleapOAuthService.java"],
  resources = ["//plugins/oauth:src/main/resources/Documentation/build.md", "//plugins/oauth:src/main/resources/Documentation/config.md", "//plugins/oauth:src/main/resources/Documentation/images/github-1.png", "//plugins/oauth:src/main/resources/Documentation/images/github-2.png", "//plugins/oauth:src/main/resources/Documentation/images/gitlab-1.png", "//plugins/oauth:src/main/resources/Documentation/images/gitlab-2.png", "//plugins/oauth:src/main/resources/Documentation/images/google-1.png", "//plugins/oauth:src/main/resources/Documentation/images/google-2.png", "//plugins/oauth:src/main/resources/Documentation/images/google-3.png", "//plugins/oauth:src/main/resources/Documentation/images/google-4.png", "//plugins/oauth:src/main/resources/Documentation/images/google-5.png"],
  deps = ["@commons-codec//jar:neverlink", "@jackson-core//jar:jar", "@jackson-databind//jar:jar", "@scribejava-apis//jar:jar", "@scribejava-core//jar:jar", "//plugins:plugin-lib-neverlink"],
)
# Rule oauth__plugin instantiated at (most recent call last):
#   /home/davido/projects/gerrit2/plugins/oauth/BUILD:10:14                                                          in <toplevel>
#   /home/davido/projects/gerrit2/tools/bzl/plugin.bzl:30:17                                                         in gerrit_plugin
#   /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/rules_java/java/defs.bzl:64:24 in java_library
# Rule java_library defined at (most recent call last):
#   /virtual_builtins_bzl/common/java/java_library.bzl:185:39 in <toplevel>
#   /virtual_builtins_bzl/common/java/java_library.bzl:170:16 in _make_java_library_rule

Currently, there are four workarounds:

  1. remove resources attribute from gerrit_plugin macro. This would corrupt plugin artifact and is not an option:
diff --git a/BUILD b/BUILD
index 81871ad..4f0b682 100644
--- a/BUILD
+++ b/BUILD
@@ -18,7 +18,6 @@ gerrit_plugin(
         "Implementation-Title: Gerrit OAuth authentication provider",
         "Implementation-URL: https://github.com/davido/gerrit-oauth-provider",
     ],
-    resources = glob(["src/main/resources/**/*"]),
     deps = [
         "@commons-codec//jar:neverlink",
         "@jackson-core//jar",
  1. Avoid using symbolik links. But we use this feature since Bazel 0.2 and would like to preserve that feature. Also note, that all Gerrit plugins that use resources attribute in gerrit_plugin macro are affected.

  2. Unflip the option: --incompatible_sandbox_hermetic_tmp, by saying: bazeldev build --noincompatible_sandbox_hermetic_tmp plugins/oauth

  3. Pass --sandbox_add_mount_pair=/tmpoption.

@fmeum
Copy link
Collaborator

fmeum commented Nov 19, 2023

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 19, 2023
@davido
Copy link
Contributor Author

davido commented Nov 19, 2023

@fmeum It seems to be related to --incompatible_sandbox_hermetic_tmp option flip per default, that was done in 7.0.0rc4.

I've updated the issue description.

@davido davido changed the title [7.0.0] Regression: singlejar doesn't follow symbolic link [7.0.0] Regression in 7.0.0rc4: singlejar doesn't follow symbolic link (after flip of --incompatible_sandbox_hermetic_tmp option per default) Nov 19, 2023
@davido davido changed the title [7.0.0] Regression in 7.0.0rc4: singlejar doesn't follow symbolic link (after flip of --incompatible_sandbox_hermetic_tmp option per default) [7.0.0] Regression in 7.0.0rc4: singlejar doesn't follow symbolic link (after flip of --incompatible_sandbox_hermetic_tmp option per default) Nov 19, 2023
@fmeum
Copy link
Collaborator

fmeum commented Nov 19, 2023

Cc @cushon

@fmeum
Copy link
Collaborator

fmeum commented Nov 19, 2023

Cc @lberki since Gerrit seems to have a working directory under /tmp.

@keertk
Copy link
Member

keertk commented Nov 19, 2023

@bazel-io fork 7.0.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Nov 19, 2023
@keertk keertk added the team-Local-Exec Issues and PRs for the Execution (Local) team label Nov 19, 2023
@cushon
Copy link
Contributor

cushon commented Nov 20, 2023

I'm not seeing strong evidence that this is a singlejar bug, vs. it accurately reporting that it's being given a path that doesn't exist when --incompatible_sandbox_hermetic_tmp is enabled.

With --noincompatible_sandbox_hermetic_tmp, singlejar see an input that is a symlink, which canonicalizes to a path outside the execroot:

EXECROOT=/usr/local/google/home/cushon/.cache/bazel/_bazel_cushon/a342130ca8fcfedda2d019bbca77cb0e/execroot
$ ls -l $EXECROOT/gerrit/plugins/oauth/src/main/resources/Documentation/images/google-5.png
-rw-r----- 1 cushon primarygroup 145177 Nov 19 08:30 $EXECROOT/gerrit/plugins/oauth/src/main/resources/Documentation/images/google-5.png
$ readlink -f $EXECROOT/gerrit/plugins/oauth/src/main/resources/Documentation/images/google-5.png
$HOME/cushon/projects/oauth/src/main/resources/Documentation/images/google-5.png
$ ls -l $EXECROOT/gerrit/plugins/oauth
lrwxrwxrwx 1 cushon primarygroup 11 Nov 19 08:30 $EXECROOT/gerrit/plugins/oauth -> ../../oauth
$ ls -l $EXECROOT/gerrit/plugins
lrwxrwxrwx 1 cushon primarygroup 53 Nov 19 11:47 $EXECROOT/gerrit/plugins -> $HOME/cushon/projects/gerrit/plugins

With --noincompatible_sandbox_hermetic_tmp

EXECROOT=/usr/local/google/home/cushon/.cache/bazel/_bazel_cushon/a342130ca8fcfedda2d019bbca77cb0e/sandbox/linux-sandbox/361/execroot
$ ls -l $EXECROOT/gerrit/plugins/oauth/src/main/resources/Documentation/images/google-5.png
lrwxrwxrwx 1 cushon primarygroup 92 Nov 20 08:13 $EXECROOT/gerrit/plugins/oauth/src/main/resources/Documentation/images/google-5.png -> /tmp/bazel-source-roots/1/plugins/oauth/src/main/resources/Documentation/images/google-5.png
# (dangling symlink?)

/tmp/bazel-source-roots/ doesn't exist at this point with --sandbox_debug.

@fmeum
Copy link
Collaborator

fmeum commented Nov 21, 2023

Thanks for the investigation!

@lberki I unfortunately won't be able to look into this in more detail this week. In order to not block the release, I could see either unflipping the flag or reverting to the state where it just disabled itself if a source root was under /tmp. What do you think?

@lberki
Copy link
Contributor

lberki commented Nov 21, 2023

I don't think this is an issue with the sources being under /tmp; rather, the issues is that the source root contains a symlink to outside of the source tree: in the above setup, projects/gerrit is the workspace root and projects/gerrit/plugins/oauth is a symlink which resolves to projects/oauth, which is not under it. So Bazel is perfectly within its rights to signal an error and thus this should be fixed on the Gerrit side somehow.

Suggestions:

  1. Use an absolute instead of a relative symlink
  2. Use a hardlink instead of a symlnk
  3. Put --noincompatible_hermetic_sandbox_tmp into the bazelrc (we're eventually going to remove this flag and then the workaround will be not to use sandboxing)

@meteorcloudy
Copy link
Member

@davido Would any of the suggestions work for you?

@davido
Copy link
Contributor Author

davido commented Nov 22, 2023

Thanks for investigation and workaround suggestions.
Yes, we can handle this.

@meteorcloudy
Copy link
Member

Thanks, then I'll close this one.

@meteorcloudy meteorcloudy closed this as not planned Won't fix, can't repro, duplicate, stale Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Local-Exec Issues and PRs for the Execution (Local) team type: bug untriaged
Projects
None yet
Development

No branches or pull requests

10 participants