rlocation in a sh_binary script fails to find requested file when bzlmod is enabled

[cgrindel]

Description of the bug:

When running a BCR presubmit test for cgrindel_bazel_starlib, the test fails due to a Bash script being unable to locate a dependent shell script.

Failed to locate cgrindel_bazel_starlib/shlib/lib/fail.sh

Example BCR PR
Buildkit Output
Shell Script

I am unable to reproduce this locally. I believe that repo mapping on the Buildkite machine is happening slightly differently than on my macOS laptop.

Below is some information on the configuration, the output after enabling RUNFILES_LIB_DEBUG, and a dump of the _repo_mapping file.

BUILD.bazel for the sh_binary

sh_binary(
    name = "buildifier",
    srcs = ["buildifier.sh"],
    data = ["@buildifier_prebuilt//buildifier"],
    visibility = ["//visibility:public"],
    deps = [
        "//shlib/lib:arrays",
        "//shlib/lib:fail",   # <-- This sh_library provides the fail.sh file.
        "@bazel_tools//tools/bash/runfiles",
    ],
)

Top of the sh_binary script:

#!/usr/bin/env bash

# --- begin runfiles.bash initialization v3 ---
# Copy-pasted from the Bazel Bash runfiles library v3.
set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
  source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
  source "$0.runfiles/$f" 2>/dev/null || \
  source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
  source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
  { echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
# --- end runfiles.bash initialization v3 ---

# DEBUG BEGIN
if [[ -f "${RUNFILES_REPO_MAPPING}" ]]; then
  echo >&2 "*** CHUCK $(basename "${BASH_SOURCE[0]}") RUNFILES_REPO_MAPPING: ${RUNFILES_REPO_MAPPING}" 
  cat >&2 "${RUNFILES_REPO_MAPPING}"
fi
# Enable debug for rlocation
export RUNFILES_LIB_DEBUG=1
# DEBUG END

# MARK - Locate Deps

fail_sh_location=cgrindel_bazel_starlib/shlib/lib/fail.sh
fail_sh="$(rlocation "${fail_sh_location}")" || \
  (echo >&2 "Failed to locate ${fail_sh_location}" && exit 1)
source "${fail_sh}"

Error Output with RUNFILES_LIB_DEBUG Enabled

INFO[runfiles.bash]: rlocation(cgrindel_bazel_starlib/shlib/lib/fail.sh): start
INFO[runfiles.bash]: runfiles_current_repository(2): caller's path is (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier)
INFO[runfiles.bash]: runfiles_current_repository(2): (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier) does not lie under the runfiles directory (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier.runfiles)
INFO[runfiles.bash]: runfiles_current_repository(2): (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier) lies in the main repository
INFO[runfiles.bash]: rlocation(cgrindel_bazel_starlib/shlib/lib/fail.sh): looking up canonical name for (cgrindel_bazel_starlib) from () in (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier.runfiles/_repo_mapping)
INFO[runfiles.bash]: rlocation(cgrindel_bazel_starlib/shlib/lib/fail.sh): canonical name of target repo is ()
ERROR[runfiles.bash]: cannot look up runfile "cgrindel_bazel_starlib/shlib/lib/fail.sh"  (RUNFILES_DIR="bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/cgrindel_bazel_starlib~override/bzlformat/tools/buildifier.runfiles", RUNFILES_MANIFEST_FILE="")
Failed to locate cgrindel_bazel_starlib/shlib/lib/fail.sh

Output of _repo_mapping File

bazel_tools,bazel_tools,bazel_tools
bazel_tools~cc_configure_extension~local_config_cc,bazel_tools,bazel_tools
bazel_tools~xcode_configure_extension~local_config_xcode,bazel_tools,bazel_tools
buildifier_prebuilt~6.0.0.1,bazel_tools,bazel_tools
buildifier_prebuilt~6.0.0.1,buildifier_prebuilt,buildifier_prebuilt~6.0.0.1
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_darwin_amd64,bazel_tools,bazel_tools
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_darwin_amd64,buildifier_darwin_amd64,buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_darwin_amd64
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_darwin_amd64,buildifier_prebuilt,buildifier_prebuilt~6.0.0.1
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_prebuilt_toolchains,bazel_tools,bazel_tools
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_prebuilt_toolchains,buildifier_darwin_amd64,buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_darwin_amd64
buildifier_prebuilt~6.0.0.1~buildifier_prebuilt_deps_extension~buildifier_prebuilt_toolchains,buildifier_prebuilt,buildifier_prebuilt~6.0.0.1
cgrindel_bazel_starlib~override,bazel_tools,bazel_tools
cgrindel_bazel_starlib~override,buildifier_prebuilt,buildifier_prebuilt~6.0.0.1
cgrindel_bazel_starlib~override,cgrindel_bazel_starlib,cgrindel_bazel_starlib~override
platforms~0.0.6,bazel_tools,bazel_tools

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I do not have a simple repro for this issue. So far, I have only been able to experience the failure when running a BCR presubmit CI test.
Example of failing test

Which operating system are you running Bazel on?

macOS, ubuntu2004

What is the output of bazel info release?

6.0.0 (on the BCR CI machine); release 6.1.0 (locally)

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

Pretty sure you are hitting #17279, which only made it into 6.1.0.

@meteorcloudy Could we raise the BCR Bazel version to 6.1.1?

[fmeum]

Now that I think about it, that's probably not the right approach.

@cgrindel Could you specify a bazel_compatibility of 6.1.0 on module?

@meteorcloudy Does Bazel CI honor the Bazel compatibility automatically?

[cgrindel]

@fmeum I added bazel_compatibility = [">=6.1.0"]. However, it appears that the BCR presubmit CI machines are all running 6.0.0.

(17:40:15) ERROR: Bazel version 6.0.0 is not compatible with module "[email protected]" (bazel_compatibility: [>=6.1.0])
--
  | (17:40:15) ERROR: Error computing the main repository mapping: Bazel compatibility check failed

Failing presubmit

[cgrindel]

BTW, I have run a test locally and verified that this issue can be reproduced on my laptop when I use 6.0.0. I think that the path forward is to upgrade the BCR presubmit CI machines and/or provide a pool with different Bazel versions.

[cgrindel]

Given that my issue is fixed in 6.1.0, I am going to close this issue. I will follow up with BCR folks to address the version of Bazel on the presubmit machines.