v0.10.0 breaks @com_google_protobuf with "cannot load '@@rules_pkg~0.10.0//:mappings.bzl': no such file"

[yuan-attrove]

My error with v0.10.0

bazel_dep(name = "rules_pkg", version = "0.10.0") gives the following error:

ERROR: error loading package '@@com_google_protobuf//': cannot load '@@rules_pkg~0.10.0//:mappings.bzl': no such file

While
bazel_dep(name = "rules_pkg", version = "0.9.1") builds fine.

Searching in the protobuf repo, it seems to be relying on v0.7.0:

Code where @rules_pkg//:mappings.bzl is used:
https://github.com/search?q=repo%3Aprotocolbuffers%2Fprotobuf%20rules_pkg&type=code

Issue #596 suggests it has been rules_pkg//pkg:mappings.bzl since v0.7.1. Not sure how the version resolution works. Why does v0.9.1 work while v0.10.0 doesn't?

My setup:

1. Use bzlmod. Have com_google_protobuf in WORKSPACE.bzlmod:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_google_protobuf",
    sha256 = "3c83e4301b968d0b4f29a0c29c0b3cde1da81d790ffd344b111c523ba1954392",
    strip_prefix = "protobuf-3.25.2",
    urls = [
        "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.2.tar.gz",
    ],
)

load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")

protobuf_deps()

2. Have a protobuf depends on "google/protobuf/timestamp.proto"

In my.proto:

import "google/protobuf/timestamp.proto";

message MyProto {
  google.protobuf.Timestamp my_time = 1;
}

In BUILD.bazel:

proto_library(
    name = "my_proto",
    srcs = [
        "my.proto",
    ],
    visibility = ["//visibility:public"],
    deps = ["@com_google_protobuf//:timestamp_proto"],
)

my_proto is further depended by a go_proto_library which is then depended by a go_binary, then further by a pkg_tar, and finally in oci_image.

bazel version:

Bazelisk version: development
Build label: 7.0.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Dec 11 16:52:42 2023 (1702313562)
Build timestamp: 1702313562
Build timestamp as int: 1702313562

[aiuto]

//:mapping.bzl has been deprecated for a long time. It should be @rules_pkg//pkg:mappings.bzl. protobuf needs a fix, or use an older rules_pkg.

…

On Thu, Jan 11, 2024 at 3:25 AM Yuan ***@***.***> wrote: My error with v0.10.0 bazel_dep(name = "rules_pkg", version = "0.10.0") gives the following error: ERROR: error loading package '@@com_google_protobuf//': cannot load '@@rules_pkg~0.10.0//:mappings.bzl': no such file While bazel_dep(name = "rules_pkg", version = "0.9.1") builds fine. Searching in the protobuf repo <https://github.com/protocolbuffers/protobuf>, it seems to be relying on v0.7.0 <https://github.com/protocolbuffers/protobuf/blob/27275da1c0033331e55277613f32457b2500e811/protobuf_deps.bzl#L113-L119> : Code where @rules_pkg//:mappings.bzl is used: https://github.com/search?q=repo%3Aprotocolbuffers%2Fprotobuf%20rules_pkg&type=code Issue #596 <#596> suggests it has been rules_pkg//pkg:mappings.bzl since v0.7.1. Not sure how the version resolution works. Why does v0.9.1 work while v0.10.0 doesn't? My setup: 1. Use bzlmod. Have com_google_protobuf in WORKSPACE.bzlmod: ***@***.***_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name = "com_google_protobuf", sha256 = "3c83e4301b968d0b4f29a0c29c0b3cde1da81d790ffd344b111c523ba1954392", strip_prefix = "protobuf-3.25.2", urls = [ "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.2.tar.gz", ], ) ***@***.***_google_protobuf//:protobuf_deps.bzl", "protobuf_deps") protobuf_deps() 1. Have a protobuf depends on "google/protobuf/timestamp.proto" In my.proto: import "google/protobuf/timestamp.proto"; message MyProto { google.protobuf.Timestamp my_time = 1; } In BUILD.bazel: proto_library( name = "my_proto", srcs = [ "my.proto", ], visibility = ["//visibility:public"], deps = ***@***.***_google_protobuf//:timestamp_proto"], ) my_proto is further depended by a go_proto_library which is then depended by a go_binary, then further by a pkg_tar, and finally in oci_image. bazel version: Bazelisk version: development Build label: 7.0.0 Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer Build time: Mon Dec 11 16:52:42 2023 (1702313562) Build timestamp: 1702313562 Build timestamp as int: 1702313562 — Reply to this email directly, view it on GitHub <#810>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHFT6EFVBM4HS7F73A3YN6OVZAVCNFSM6AAAAABBWCMPN6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3TMMBTGIYTANI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[yuan-attrove]

//:mapping.bzl has been deprecated for a long time.

Based on 596, I assume it was deprecated before v0.9.1. If that's true it doesn't explain why v0.9.1 works while v0.10.0 doesn't.

I have created a repo for reproducing the issue: https://github.com/yuan-attrove/rules_pkg_810.

From the root of the repo, the command

bazelisk build :server-image

works as the repo is.

Changing 0.9.1 to 0.10.0 makes the above command fail with the error in comment#1 (#810 (comment))

protobuf needs a fix, or use an older rules_pkg.

protobuf seems to pin rules_pkg to v0.7.0: https://github.com/protocolbuffers/protobuf/blob/27275da1c0033331e55277613f32457b2500e811/protobuf_deps.bzl#L113-L119. But again I am not sure how version resolution works here.

[yuan-attrove]

A side note, may not be relevant. 0.10.0 is not showing up in bazel central registry: https://registry.bazel.build/modules/rules_pkg

[aiuto]

The module seems to be there: https://github.com/bazelbuild/bazel-central-registry/tree/main/modules/rules_pkg/0.10.0

…

On Thu, Jan 11, 2024 at 3:00 PM Yuan ***@***.***> wrote: A side note, may not be relevant. 0.10.0 is not showing up in bazel central registry: https://registry.bazel.build/modules/rules_pkg — Reply to this email directly, view it on GitHub <#810 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHQBMG5YPLQCQ7Y5QDYOBAHRAVCNFSM6AAAAABBWCMPN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBXHA3TKMZZHA> . You are receiving this because you commented.Message ID: ***@***.***>

[alokpr]

protobuf is indeed using an old version - 0.7.0 - https://github.com/protocolbuffers/protobuf/blob/7b42f1c08b534f9e43629284e22acce6b13e7136/protobuf_deps.bzl#L116

However, this is still a problem for a project that uses the latest version of rules_pkg and protobuf. Would it be possible to provide a shim layer for backwards compatibility (like deps.bzl)?

[aiuto]

> Would it be possible to provide a shim layer for backwards compatibility (like deps.bzl)?

That would not be cost effective work from my point of view. You can fix this in your WORKSPACE or MODULE.bazel by loading the newer rules_pkg before protobuf. If protobuf does not work with current rules_pkg (which is doubtful), then you should try to upgrade the protobuf you use version by version until they work together. In general, relying on bzlmod or deps.bzl to resolve to the set of things you actually need is a game of whack-a-mole. You really need to test and certify suites of tools together.

[alokpr]

That's exacty what we are doing - loading rules_pkg before protobuf. The latest version of protobuf does not work against the latest version of rules_pkg. It uses @rules_pkg//:mappings.bzl which has been deleted in rules_pkg.

https://github.com/protocolbuffers/protobuf/blob/7b42f1c08b534f9e43629284e22acce6b13e7136/BUILD.bazel#L5

[aiuto]

So, splice in mappings.bzl as a patch. Or, copy rules_pkg in to your third_party tree, make it a local repository and add a small .bzl file which points to the real one.

…

On Sun, Jan 14, 2024 at 3:12 PM Alok Priyadarshi ***@***.***> wrote: That's exacty what we are doing - loading rules_pkg before protobuf. The latest version of protobuf does not work against the latest version of rules_pkg. It uses @rules_pkg//:mappings.bzl which has been deleted in rules_pkg. https://github.com/protocolbuffers/protobuf/blob/7b42f1c08b534f9e43629284e22acce6b13e7136/BUILD.bazel#L5 — Reply to this email directly, view it on GitHub <#810 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHE365K3QGRAY43EPYTYOQ3ZDAVCNFSM6AAAAABBWCMPN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRGA3DAMRQGU> . You are receiving this because you commented.Message ID: ***@***.***>

[alokpr]

Wouldn't all clients of rules_pkg have to do this? Wouldn't it be better to add this shim in rules_pkg itself to make the upgrade to a new version of rules_pkg easier? I suspect that most projects would just pin rules_pkg to an old version to avoid this breaking change.

[aiuto]

Wouldn't all clients of rules_pkg have to do this?

Or, someone could fix protobuf so it can use a modern one.

…

On Sun, Jan 14, 2024 at 9:44 PM Alok Priyadarshi ***@***.***> wrote: Wouldn't all clients of rules_pkg have to do this? Wouldn't it be better to add this shim in rules_pkg itself to make the upgrade to a new version of rules_pkg easier? I suspect that most projects would just pin rules_pkg to an old version to avoid this breaking change. — Reply to this email directly, view it on GitHub <#810 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBJACN3VNHIDHXRSADYOSJZDAVCNFSM6AAAAABBWCMPN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRGIYTQMBYGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[phlax]

#596 (comment)

[phlax]

updating protobuf is non-trivial due to rules_ruby having a dependency on this repo also

ill look at it again next week

[aiuto]

I'll add in a symlink for mappings.bzl, and do a patch release, but this must also be fixed in protobuf.
protocolbuffers/protobuf#15779

[aiuto]

Being addressed in #817

[aiuto]

Fixed by #817. To be in 0.10.1. That is done, but not in BCR yet.