repository_ctx.download doesn't use netrc

[dmivankov]

Description of the problem / feature request:

It seems that repository_ctx.download(url) doesn't handle netrc auth (while http_archive/http_file can use it, also wget can fetch same url).

Rule in question is jvm_maven_import_external https://github.com/bazelbuild/bazel/blob/master/tools/build_defs/repo/jvm.bzl#L109

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Don't have repro yet, will try to add later

What operating system are you running Bazel on?

NixOS

What's the output of bazel info release?

release 4.1.0- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

nixpkgs

[adam-azarchs]

It doesn't out of the box, but it can be made to with a bit of boiler-plate:

load(
    "@bazel_tools//tools/build_defs/repo:utils.bzl",
    "read_netrc",
    "use_netrc",
)
load("@bazel_skylib//lib:versions.bzl", "versions")

def _use_netrc(netrc, urls, patterns):
    if versions.is_at_most("3.0.0", versions.get()):
        return use_netrc(netrc, urls)
    return use_netrc(netrc, urls, patterns)

def get_auth(ctx, urls):
    """Given the list of URLs obtain the correct auth dict.
    Args:
        ctx: The repository context.
        urls: A list of URLs.
    Returns:
        A dictionary of workspace attributes to update.
    """

    # Mostly copied from https://github.com/bazelbuild/bazel/blob/3.7.2/tools/build_defs/repo/http.bzl
    if ctx.attr.netrc:
        netrc = read_netrc(ctx, ctx.attr.netrc)
        return _use_netrc(netrc, urls, ctx.attr.auth_patterns)

    if not ctx.os.name.startswith("windows"):
        if "HOME" in ctx.os.environ:
            netrcfile = "%s/.netrc" % (ctx.os.environ["HOME"],)
            if ctx.execute(["test", "-f", netrcfile]).return_code == 0:
                netrc = read_netrc(ctx, netrcfile)
                return _use_netrc(netrc, urls, ctx.attr.auth_patterns)
    elif "USERPROFILE" in ctx.os.environ and ctx.os.name.startswith("windows"):
        netrcfile = "%s/.netrc" % (ctx.os.environ["USERPROFILE"])
        if ctx.path(netrcfile).exists:
            netrc = read_netrc(ctx, netrcfile)
            return _use_netrc(netrc, urls, ctx.attr.auth_patterns)
        netrcfile = "%s/_netrc" % (ctx.os.environ["USERPROFILE"])
        if ctx.path(netrcfile).exists:
            netrc = read_netrc(ctx, netrcfile)
            return _use_netrc(netrc, urls, ctx.attr.auth_patterns)

    return {}
    
def _my_rule_implementation(ctx):
    auth = get_auth(ctx, all_urls)

    download_info = ctx.download_and_extract(
        ctx.attr.urls,
        ctx.attr.sha256,
        auth = auth,
    )

That said it isn't ideal for every rule implementation to have to do this.

[pauldraper]

In recent versions, it's even simpler:

load("@bazel_tools//tools/build_defs/repo:utils.bzl", "read_netrc", "read_user_netrc", "use_netrc")

def _get_auth(ctx, urls):
    """Given the list of URLs obtain the correct auth dict."""
    if ctx.attr.netrc:
        netrc = read_netrc(ctx, ctx.attr.netrc)
    else:
        netrc = read_user_netrc(ctx)
    return use_netrc(netrc, urls, ctx.attr.auth_patterns)

def _my_rule_implementation(ctx):
    auth = _get_auth(ctx, all_urls)

    download_info = ctx.download_and_extract(
        ctx.attr.urls,
        ctx.attr.sha256,
        auth = auth,
    )

my_rule = rule(
  attrs = {
    "auth_patterns": attr.string_dict(),
    "netrc": attr.string(),
    "urls": attr.string_list(),
  },
  implementation = _my_rule_implementation,
)

[aryeh-looker]

In recent versions, it's even simpler:

load("@bazel_tools//tools/build_defs/repo:utils.bzl", "read_netrc", "read_user_netrc", "use_netrc")

def _get_auth(ctx, urls):
    """Given the list of URLs obtain the correct auth dict."""
    if ctx.attr.netrc:
        netrc = read_netrc(ctx, ctx.attr.netrc)
    else:
        netrc = read_user_netrc(ctx)
    return use_netrc(netrc, urls, ctx.attr.auth_patterns)

def _my_rule_implementation(ctx):
    auth = _get_auth(ctx, all_urls)

    download_info = ctx.download_and_extract(
        ctx.attr.urls,
        ctx.attr.sha256,
        auth = auth,
    )

my_rule = rule(
  attrs = {
    "auth_patterns": attr.string_dict(),
    "netrc": attr.string(),
    "urls": attr.string_list(),
  },
  implementation = _my_rule_implementation,
)

I would have never found this in a billion years. Any link to documentation or your process on how you figured this out?

[adam-azarchs]

Everything in @bazel_tools//, including the built-in http_archive, lives in https://github.com/bazelbuild/bazel/tree/6.1.2/tools/. You can just read the code there. Documentation would be nice, though.

[pauldraper]

The always-comprehensive, always-accurate documentation: the Source.

:/

[Wyverald]

Note for anyone blocked on this: using the NETRC environment variable or the --credential_helper flag should work around the issue, until we implement a proper fix. See integration test at

bazel/src/test/py/bazel/bzlmod/bzlmod_credentials_test.py

Line 135 in 62799ed

def testCredentialsFromNetrc(self):

for an example.