Label resolution fails by repo mapping when used by a repo rule inside a module extension

[fitzner]

Description of the problem / feature request:

We are facing an issue when a module is used from a bzlmod registry and e.g. a http_archive repo rule inside the extension do reference a label from the module repository.

It seems that the repo mapping is wrongly used in that context.

build@efa2cf645951:/workspace/example$ bazelisk build @foo//:test
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Repository foo.0.1.0.external_http.foo_external1 instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/bazel_tools/tools/build_defs/repo/http.bzl:364:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'foo.0.1.0.external_http.foo_external1':
   Traceback (most recent call last):
	File "/home/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121, column 28, in _http_archive_impl
		workspace_and_buildfile(ctx)
	File "/home/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 61, column 41, in workspace_and_buildfile
		ctx.file("BUILD.bazel", ctx.read(ctx.attr.build_file))
Error in read: Unable to load package for @foo//:BUILD.bug: The repository '@foo' could not be resolved: Repository '@foo' is not defined
ERROR: <builtin>: fetching http_archive rule //external:foo.0.1.0.external_http.foo_external1: Traceback (most recent call last):
	File "/home
[example.tar.gz](https://github.com/bazelbuild/bazel/files/7953774/example.tar.gz)
/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121, column 28, in _http_archive_impl
		workspace_and_buildfile(ctx)
	File "/home/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 61, column 41, in workspace_and_buildfile
		ctx.file("BUILD.bazel", ctx.read(ctx.attr.build_file))
Error in read: Unable to load package for @foo//:BUILD.bug: The repository '@foo' could not be resolved: Repository '@foo' is not defined
ERROR: /home/build/.cache/bazel/_bazel_build/56512a070a25878a45bf0c1a46021ad9/external/foo.0.1.0/BUILD:3:10: @foo.0.1.0//:test depends on @foo.0.1.0.external_http.foo_external1//:bar in repository @foo.0.1.0.external_http.foo_external1 which failed to fetch. no such package '@foo.0.1.0.external_http.foo_external1//': Unable to load package for @foo//:BUILD.bug: The repository '@foo' could not be resolved: Repository '@foo' is not defined
ERROR: Analysis of target '@foo.0.1.0//:test' failed; build aborted: Analysis failed
INFO: Elapsed time: 4.213s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (4 packages loaded, 5 targets configured)

We just observe the issue when the module is fetched from a registry. When we use 'archive_override' or 'local_path_override' in the main module everything works.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

MODULE.bazel in the foo module and registry

module(
    name = "foo",
    version = "0.1.0",
    compatibility_level = 0,
    toolchains_to_register = [
    ],
)

external_http = use_extension("@foo//:extensions.bzl", "external_http")
use_repo(external_http, "foo_external1")

source.json from the registry

{
    "integrity": "sha256-TcOolQZlJ56RcXC8yrPbJr6plC56Lc7D1I4B0bNyuNU=",
    "url": "http://localhost:51213/foo/0.1.0/foo-0.1.0.tar.gz"
}

extensions.bzl in the foo module

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

def _external_http_impl(ctx):
    http_archive(
        name = "foo_external1",
        urls = [
            "https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz"
        ],
        sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
        build_file = "@foo//:BUILD.bug",
    )

external_http = module_extension(
    implementation = _external_http_impl,
    tag_classes = {
    },
)

BUILD.bug from in th foo module

package(default_visibility = ["//visibility:public"])

filegroup(
    name = "bar",
    srcs = [
    ],
)

MODULE.bazel in the test project

module(
    name = "main",
    version = "0.1",
)

bazel_dep(name = "foo", version = "0.1.0")

# Resolution of BUILD.bug from http_archive is ok
#archive_override(
#    module_name = "foo",
#    urls = "http://localhost:51213/foo/0.1.0/foo-0.1.0.tar.gz",
#    integrity = "sha256-TcOolQZlJ56RcXC8yrPbJr6plC56Lc7D1I4B0bNyuNU="
#)

# Resolution of BUILD.bug from http_archive is ok
#local_path_override(
#    module_name = "foo",
#    path = "../projects/foo/"
#)

When bazelisk build @foo//:test is executed it leads to the described error.

What operating system are you running Bazel on?

Linux Ubuntu 20.04.3, MacOs 10.15.7

What's the output of bazel info release?

release 5.0.0

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Replace this line with your answer.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Replace this line with your answer.

Have you found anything relevant by searching the web?

• Bzlmod: stardoc is broken by repo mappings #14140

Any other information, logs, or outputs that you want to share?

Find the example package to reproduce the issue as an attachment.

example.tar.gz

[Wyverald]

Thank you @fitzner for the extensive repo archive! I'm looking into this.

[Wyverald]

Sorry for the very late response but I finally found some cycles to debug this. Turns out this is a very subtle bug in our usage of PackageFactory.newExternalPackageBuilder which causes us to treat all Bzlmod repo rules to be created as if for the package //external.

This means that all text labels given to attributes are treated as if they're specified in the imaginary //external package, which in turn means that "@foo//:bar" will get passed through the default empty repo mapping (not even the main repo's mapping!), and even worse, something written as ":my_target" will get interpreted as "@//external:my_target".

Unfortunately the fix here isn't trivial, since it's not very clear what to use instead of //external, and even if we decided on some meaningful value for this package ID, we often do equality checks with //external to decide whether a package is being built for repo rules instead of normal build rules (see for example

bazel/src/main/java/com/google/devtools/build/lib/packages/Package.java

Lines 1086 to 1088 in c878c67

	boolean isWorkspace() {
	return pkg.getPackageIdentifier().equals(LabelConstants.EXTERNAL_PACKAGE_IDENTIFIER);
	}

)

So this change might take a little more time. At least we know what's broken!

[aaronmondal]

We just switched to bzlmod and encountered this.

While this bug breaks several attributes in http_archive, it should be possible to work around most of those. For instance, build_file is broken, but build_file_content should work. Same for patches and patch_cmds.

[nachumg]

Is this the same as the following scenario?

Load repo "a" with:

• patch "@x//some/path:some_target"
• repo_mapping "@x": "@foo"

When I build, I expect bazel to rewrite the patch to @foo//some/path:some_target
But it does not.

[Wyverald]

Could you clarify your example with a code snippet? It's unclear to me if those are repo rule attributes, if this is a WORKSPACE file or a .bzl file or some other file, etc.

In general (assuming this is the WORKSPACE file), the repo mapping you specify for a repo applies to things within the repo, not the patch file you're passing to it (which is resolved in the WORKSPACE file's context).

[nachumg]

WORKSPACE:

workspace(name = "my_project")

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") 

git_repository(
  name = "a",
  remote = "<some-URL>"
  branch = "main",
  patches = [
    "@x//:constants.patch",  # Should be rewritten to: "@patch_repo//:constants.patch"
  ],
  repo_mapping = {
    '@x': '@patch_repo',  # Should cause the abovementioned rewrite.
  }
)

[Wyverald]

Then that is intended (see my comment above). Labels within the a repo will be remapped, but the WORKSPACE file is not within the a repo, so @x won't be mapped to @patch_repo.

[nachumg]

Thanks. What should be done for this dep tree (repo-name:version-number):

a:1 -> b:2 with patch @a//some:file
a:3 -> b:2 with patch @a//some:file

Where multiple versions of a and b are explicitly allowed.

In this case we need two instantions of b:

1. b:2 patched with some:file from a:1
2. b:2 patched with some:file from a:3

[nachumg]

Hi, ping re my q above. Thanks.

[nachumg]

Ping again.

[Wyverald]

sorry for the delay. In your case, since you need two instantions of b, you'll necessarily need to have two b:2's, each with a different name. Then presumably you'll need to specify repo mappings for a:1 and a:3, respectively. Depending on how your repo names are structured, you may want:

• for repo a:1, specify repo_mappings = {"@b": "@b_2_with_a_1_patch"}, and for repo a:3, specify repo_mappings = {"@b": "@b_2_with_a_3_patch"}
• OR, if you want to use labels like @b_2//:whatever in a repos, replace @b with @b_2 above.

[nachumg]

Thanks. This is exactly what we do for our h/ware IPs, using our dep-management tool.