Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MODULE.bazel doesn't define @maven repository #17176

Closed
comius opened this issue Jun 18, 2024 · 9 comments · Fixed by #17402
Closed

MODULE.bazel doesn't define @maven repository #17176

comius opened this issue Jun 18, 2024 · 9 comments · Fixed by #17402
Labels
untriaged auto added to all issues by default when created.

Comments

@comius
Copy link
Contributor

comius commented Jun 18, 2024

What version of protobuf and what language are you using?

Version: 27.1 (and at head)
Language: Bazel 7+ with bzlmod enabled

What did you do?

Steps to reproduce the behavior:

$ bazel --version
bazel 7.2.0
$ touch WORKSPACE.bzlmod   # disables WORKSPACE file for bzlmod
$ bazel build --enable_bzlmod //java/util

What did you expect to see
Successful build.

What did you see instead?

ERROR: no such package '@@[unknown repo 'maven' requested from @@]//': The repository '@@[unknown repo 'maven' requested from @@]' could not be resolved: No repository visible as '@maven' from main repository
ERROR: /usr/local/google/home/ilist/src/protobuf/java/util/BUILD.bazel:8:13: no such package '@@[unknown repo 'maven' requested from @@]//': The repository '@@[unknown repo 'maven' requested from @@]' could not be resolved: No repository visible as '@maven' from main repository and referenced by '//java/util:util'
ERROR: Analysis of target '//java/util:util' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.359s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

Anything else we should know about your project / environment

This problem practically prevents protobuf to be used as a transitive dependency with bzlmod.

@comius comius added the untriaged auto added to all issues by default when created. label Jun 18, 2024
@comius
Copy link
Contributor Author

comius commented Jun 18, 2024

cc @alexeagle @fmeum @meteorcloudy

@meteorcloudy
Copy link
Contributor

I also hit this in: bazelbuild/bazel#22719

One question is, should grpc use the default @maven repo so that it uses the same set of dependencies as the root project or it should define its own. /cc @shs96c

@meteorcloudy
Copy link
Contributor

Another question is should protobuf users even depend on //java/util? Or they should just fetch the protobuf jars with rules_jvm_external?

@comius
Copy link
Contributor Author

comius commented Jun 18, 2024

Users should use java_proto_library and not the //java/util directly.

But protobuf with bzlmod doesn’t work because it needs //java/util for itself.

I’ve hit the problem migrating Bazel to latest protobuf repo

@shs96c
Copy link

shs96c commented Jun 18, 2024

@meteorcloudy, the rule of thumb is "the only time a ruleset should contribute to the maven workspace is iff the user has to use these specific versions in their own apps"

In contrib_rules_jvm, we need to link the test runner against junit5, but we don't want to force a user to pick the version we do, so we jump through a few hoops to make sure we don't expose the junit5 dep to the user, and then require them to add the junit deps to their own tests. At runtime, we sniff for required dependencies, so we can offer a helpful error message.

A similar approach for things like protobuf (and presumably java-grpc) of asking people to manually add a single maven coordinate to their own project would be the ideal thing to do.

@shs96c
Copy link

shs96c commented Jun 18, 2024

IME, people who are depending on libraries that depend on protobuf often already have a copy of the required jar on their classpath.

@alexeagle
Copy link
Contributor

I think it's a design mistake for a repo to provide both a Bazel module and language-specific libraries. I suggest dropping Maven and let the user give the label of the protobuf runtime like toolchains_protoc.
I can give more details if you're interested

@comius
Copy link
Contributor Author

comius commented Jun 18, 2024

I think it's a design mistake for a repo to provide both a Bazel module and language-specific libraries. I suggest dropping Maven and let the user give the label of the protobuf runtime like toolchains_protoc. I can give more details if you're interested

I don’t see how this is relevant, because the @maven dependencies are external to protoc. Delete workspace file and you can’t build a runtime .jar in this case. So alternatives are to get guava and other libraries form another bzlmod repo or vendor them. I think using maven is a good approach here.

It’s certainly a design decision to have rules and toolchains together in the same module/repository. Let’s discuss this somewhere else and not in a bug report.

@joca-bt
Copy link

joca-bt commented Jun 20, 2024

Using the name @maven can result in surprising errors for users, see bazel-contrib/rules_jvm_external#1168 for an example. I think in such situations the repo should use a different name for the maven repository to avoid clashing with users.

mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

Signed-off-by: Mike Bland <[email protected]>
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 3, 2024
Updates as much as possible except rules_jvm_external.

The latest rules_jvm_external v6.1 breaks this project, both under the
previous Bazel version (7.0.2) and the new one (7.2.1). I've filed
bazel-contrib/rules_jvm_external#1189, which uses this repo and this PR as
an example.

See the last section below for details.

---

Bazel update:
- Bazel v7.2.1
  https://github.com/bazelbuild/bazel/releases/tag/7.2.1

Bazel module updates:
- bazel-skylib v1.7.1
  https://github.com/bazelbuild/bazel-skylib/releases/tag/1.7.1
- platforms v0.0.10
  https://github.com/bazelbuild/platforms/releases/tag/0.0.10
- rules_proto v6.0.2
  https://github.com/bazelbuild/rules_proto/releases/tag/6.0.2

JAR updates:
- com.google.code.gson:gson:2.11.0
- com.google.guava:guava:33.2.1-jre
- commons-cli:commons-cli:1.8.0

Test JAR updates:
- com.google.googlejavaformat:google-java-format:1.22.0
- com.google.truth:truth:1.4.3
- com.google.truth.extensions:truth-java8-extension:1.4.3
- org.mockito:mockito-core:5.12.0

---

Added explicit module spec and repinned the maven deps for:
- protobuf v27.2
  https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

Adding protobuf explicitly resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

---

rules_jvm_external v6.1 somehow creates duplicate `jvm_import` rules for
binary and source jars, instead of using the `srcjar` attribute:

```txt
ERROR: Traceback (most recent call last):
  File ".../external/rules_jvm_external~~maven~maven/BUILD",
    line 34, column 11, in <toplevel>
      jvm_import(

Error in jvm_import: jvm_import rule
  'com_google_auto_value_auto_value_annotations' in package ''
  conflicts with existing jvm_import rule, defined at
  .../external/rules_jvm_external~~maven~maven/BUILD:9:11
```

The content of rules_jvm_external~~maven~maven/BUILD at lines 9 and 34:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
```

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar"],
```

This pattern repeats for all the JAR targets.

The BUILD contents from v5.3, which builds successfully both before and
after applying the PR changes:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
  srcjar = "com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar",
```

---------

Signed-off-by: Mike Bland <[email protected]>
copybara-service bot pushed a commit that referenced this issue Jul 11, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2.

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 11, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 11, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 12, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 12, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 12, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
copybara-service bot pushed a commit that referenced this issue Jul 16, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 651013133
haberman pushed a commit that referenced this issue Sep 18, 2024
Create WORKSPACE.bzlmod. Before building with Bzlmod resulted in use of full WORKSPACE.
Some repos are still there, but the file should eventually be empty.

Add dep to rules_kotlin 1.9.0. This was the first version available on BCR. It pushed upgrade of
rules_jvm_external to 6.0 and rules_java to 6.5.2 (keep 6.0.0 on Bazel 6.3.0).

Add missing maven and other deps to MODULE.bazel

CI changes:
Disable Bazel 6.4.0 with bzlmod. rules_jvm_external 6.0 use use_repo_rule, which is not supported by Bazel 6.
Add C++ build "Bazel7 with Bzlmod" enabled.
Add Java builds with "Bazel 7 with/without Bzlmod".

Fixes: #17176
PiperOrigin-RevId: 652773197
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
untriaged auto added to all issues by default when created.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants