401 Unauthorized during oci_pull

[lbjcom]

I am migrating my project from rules_docker to rules_oci. During the migration I found that oci_pull raises 401 Unauthorized in following case:

$ bazel build //:image
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
INFO: Repository tritonserver_22.08_single instantiated at:
  /home1/irteam/bong/test/WORKSPACE:26:9: in <toplevel>
  /home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/pull.bzl:133:18: in oci_pull
Repository rule oci_pull defined at:
  /home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl:434:27: in <toplevel>
WARNING: Download from https://nvcr.io/v2/nvidia/tritonserver/manifests/sha256:98d08ad85f789fc0ca67c1a0e5db1356d35f096f82446df336e2bcc8fe65f1ea failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 401 Unauthorized
ERROR: An error occurred during the fetch of repository 'tritonserver_22.08_single':
   Traceback (most recent call last):
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 357, column 46, in _oci_pull_impl
                mf, mf_len = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 280, column 74, in lambda
                download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 255, column 18, in _download_manifest
                _download(
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 223, column 23, in _download
                return download_fn(
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/download.bzl", line 83, column 35, in _download
                command.extend(_auth_to_header(url, auth))
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/download.bzl", line 9, column 24, in _auth_to_header
                if auth_val["type"] == "pattern":
Error: key "type" not found in dictionary
ERROR: /home1/irteam/bong/test/WORKSPACE:26:9: fetching oci_pull rule //external:tritonserver_22.08_single: Traceback (most recent call last):
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 357, column 46, in _oci_pull_impl
                mf, mf_len = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 280, column 74, in lambda
                download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 255, column 18, in _download_manifest
                _download(
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/pull.bzl", line 223, column 23, in _download
                return download_fn(
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/download.bzl", line 83, column 35, in _download
                command.extend(_auth_to_header(url, auth))
        File "/home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/rules_oci/oci/private/download.bzl", line 9, column 24, in _auth_to_header
                if auth_val["type"] == "pattern":
Error: key "type" not found in dictionary
ERROR: /home/bongjinlee/.cache/bazel/_bazel_bongjinlee/9b03f7358921bbbb8549beadd376cf6a/external/tritonserver_22.08/BUILD.bazel:1:6: @tritonserver_22.08//:tritonserver_22.08 depends on @tritonserver_22.08_single//:tritonserver_22.08_single in repository @tritonserver_22.08_single which failed to fetch. no such package '@tritonserver_22.08_single//': key "type" not found in dictionary
ERROR: Analysis of target '//:image' failed; build aborted:
INFO: Elapsed time: 0.476s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 7 targets configured)

with following WORKSPACE and BUILD files:

• WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_oci",
    sha256 = "db57efd706f01eb3ce771468366baa1614b5b25f4cce99757e2b8d942155b8ec",
    strip_prefix = "rules_oci-1.0.0",
    url = "https://github.com/bazel-contrib/rules_oci/releases/download/v1.0.0/rules_oci-v1.0.0.tar.gz",
)

load("@rules_oci//oci:dependencies.bzl", "rules_oci_dependencies")

rules_oci_dependencies()

load("@rules_oci//oci:repositories.bzl", "LATEST_CRANE_VERSION", "LATEST_ZOT_VERSION", "oci_register_toolchains")

oci_register_toolchains(
    name = "oci",
    crane_version = LATEST_CRANE_VERSION,
    # Uncommenting the zot toolchain will cause it to be used instead of crane for some tasks.
    # Note that it does not support docker-format images.
    # zot_version = LATEST_ZOT_VERSION,
)

load("@rules_oci//oci:pull.bzl", "oci_pull")

oci_pull(
    name = "tritonserver_22.08",
    image = "nvcr.io/nvidia/tritonserver",
    # tag = "22.08-py3",
    digest = "sha256:98d08ad85f789fc0ca67c1a0e5db1356d35f096f82446df336e2bcc8fe65f1ea",
)

• BUILD

load("@rules_oci//oci:defs.bzl", "oci_image", "oci_tarball")

oci_image(
    name = "image",
    base = "@tritonserver_22.08",
    entrypoint = [ "main.py" ],
)

Please let me know how I can resolve this problem 😢

Thanks!

[prestonvanloon]

I have this same problem as well.

The problem is here:

rules_oci/oci/private/download.bzl

Line 9 in e426977

if auth_val["type"] == "pattern":

Probably just needs to check if that key is in the dictionary before calling it.

[lbjcom]

@prestonvanloon

It seems like your patch is applied to 1.2.0(release note) but this error still happens to me 😢 even though I update rules_oci to 1.2.0.

Does 1.2.0 work for you?

[iDiogenes]

@lbjcom - It works for me with this release, what does your oci_pull() look like?

[prestonvanloon]

@lbjcom no -- it doesn't work for me still. I don't have this fatal error anymore, but i am trying to oci_pull on a machine with no authorization credentials whatsoever and that doesn't work. rules_oci is pulling with curl when no credentials are present and then that is giving a different sha256.

Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 9b8e0854865dcaf49470b4ec305df45957020fbcf17b71eeb50ffd3bc5bf885d

If #292 is resolved, it might help. I'm not sure.

Edit: Link to public CI run: https://buildkite.com/prysmatic-labs/prysm/builds/74812#018955ea-40fe-4419-afb3-f4d3c2f627ce

[goniz]

I'm experiencing the same error as @prestonvanloon and I think that it is related to the option '--experimental_remote_downloader' we both have enabled.
It some how breaks the auth as far as I managed to figure out

[lbjcom]

@iDiogenes Here's the oci_pull(). I am just trying to pull tritonserver image from nvidia ngc.

oci_pull(
    name = "tritonserver_22.08",
    # image = "nvcr.io/nvidia/tritonserver",
    registry = "nvcr.io",
    repository = "nvidia/tritonserver",
    tag = "22.08-py3",
    # digest = "sha256:98d08ad85f789fc0ca67c1a0e5db1356d35f096f82446df336e2bcc8fe65f1ea",
)

[HDYA]

@iDiogenes Here's the oci_pull(). I am just trying to pull tritonserver image from nvidia ngc.

oci_pull(
    name = "tritonserver_22.08",
    # image = "nvcr.io/nvidia/tritonserver",
    registry = "nvcr.io",
    repository = "nvidia/tritonserver",
    tag = "22.08-py3",
    # digest = "sha256:98d08ad85f789fc0ca67c1a0e5db1356d35f096f82446df336e2bcc8fe65f1ea",
)

maybe need to add NVCR to _WWW_AUTH in oci/private/pull.bzl?