Code generated w/ protobuf-java 3.22.0 broken in Quarkus

[snazy]

Code that is generated with the new protobuf release 3.22.0 does not work in Quarkus (basically all current releases).

The exception looks somewhat like this:

Caused by: java.lang.IllegalAccessError: class <some-generated-protobuf-code> tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (<some-generated-protobuf-code> is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @4f5c30b1; com.google.protobuf.LazyStringArrayList is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @6d4a65c6)

protobuf code generated w/ versions < 3.22.0 referenced the public static field com.google.protobuf.LazyStringArrayList#EMPTY, the static method emptyList() is package-protected before 3.22.0.

In 3.22.0 emptyList() became public static and code generated w/ 3.22.0 references that method.

However, the Quarkus bom enforces protobuf-java to 3.21.9, so code generated w/ 3.22.0 won't be able to access the package-protected method emptyList().

Would it be possible to let the generated code reference the EMPTY field at least for a while before switching to emptyList()?

Opened an issue on the Quarkus as well: quarkusio/quarkus#31240

[deannagarcia]

I see quarkus upgraded to java 3.22.0, did that solve your issue?

[piotrantosik]

hi, I have same error without quarkus, just generating code from protobuf in Kotlin app:

error: emptyList() is not public in LazyStringArrayList; cannot be accessed from outside package com.google.protobuf.LazyStringArrayList.emptyList();

packages versions:

    api(group = "io.grpc", name = "grpc-protobuf", version = "1.53.0")
    api(group = "io.grpc", name = "grpc-stub", version = "1.53.0")
    api(group = "com.google.protobuf", name = "protobuf-kotlin", version = "3.22.0")
    api(group = "io.grpc", name = "grpc-kotlin-stub", version = "1.3.0")

proto file:

    repeated string values = 1;

[googleberg]

@snazy @piotrantosik If I understand correctly, you are creating gencode with protoc 3.22 and attempting to use that with a runtime that predates the 3.22 release? If that is the case, unfortunately, that is not a supported configuration. Gencode from any protobuf release can only be used with its contemporary runtime or a later runtime within the same major version.

If you need to use an older protobuf runtime, you'll need to generate your proto bindings using the protoc from that same release.

[snazy]

Yes, that's exactly the situation. code-gen w/ 3.22.0 and runtime on 3.21.9 (forced dependency).

I see the "supported combination constraint". OTOH it's a tricky situation for users - providing generated classes, but runtimes can do "their own thing" and (forcefully) pin dependencies.

A workaround (that I don't like in general), would be to let people build uber-jars with a relocated protobuf runtime. But then, beside that it's unnecessarily complicated, you get into ugly situations, that users of your shaded runtime cannot "just" bump the runtime (there might be bugfixes or security constraints).

Of course, there is a very tight relationship between the generated code and the runtime (no doubt, it's necessary).
However there are necessarily breaking changes and changes that don't need to break things. In this case (removing the public modifier) it wouldn't have to be a breaking change.

The bump to 3.22.0 was merged for Quarkus (not yet released).

[googleberg]

We'd love to make this easier but we end up with requests for constraints in both directions:

1. Old gencode + new runtime
2. New gencode + old runtime

If we satisfy both, we can never clean up anything and certain fixes cannot be done.
We settled on supporting old gencode with new runtimes in the same major version line since that enables forward progress while still allowing proto bindings to be pre-compiled as a part of library APIs.

New gencode with old runtimes really doesn't make as much sense from a support standpoint. It becomes very difficult to reason about what functionality is actually available to the gencode. In most cases, it is ok to stick with a slightly stale protoc for gencode until the runtime can be updated for all the components in the system.

Over the next year or so, we plan to formalize this with a minimum required runtime check in the gencode -- this will cause a breakage whenever there is an attempt to use new gencode with an old runtime (rather than having it appear to work only to fail in mysterious ways later). Mismatches are simply too much of a support burden.

Sorry for the friction, but hopefully this information can be used to plan going forward.

[ejona86]

FYI: grpc-java is looking at downgrading to 3.21 to avoid this issue. grpc/grpc-java#10202

[ejona86]

nm. I had mis-read what was the codegen and runtime versions. Yes, codegen version <= runtime version. And as what may be the true source of pain here, compiling protobuf-java <= runtime protobuf-java.

[mdetemad]

I am facing the same issue:

java.lang.IllegalAccessError: tried to access method com.google.protobuf.LazyStringArrayList.emptyList()Lcom/google/protobuf/LazyStringArrayList; from class com.google.iam.v1.TestIamPermissionsRequest
	at com.google.iam.v1.TestIamPermissionsRequest.<init>(TestIamPermissionsRequest.java:127)
	at com.google.iam.v1.TestIamPermissionsRequest.<clinit>(TestIamPermissionsRequest.java:918)
	at com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.<clinit>(GrpcKeyManagementServiceStub.java:416)

with these packages:

    "io.grpc:grpc-api:1.59.0",
    "io.grpc:grpc-inprocess:1.59.0",
    "io.grpc:grpc-testing:1.59.0",
    "com.google.api-client:google-api-client:2.2.0",
    "com.google.api:gax:2.35.0",
    "com.google.api:gax-grpc:2.35.0",
    "com.google.auth:google-auth-library-oauth2-http:1.20.0",
    "com.google.cloud:google-cloud-kms:2.31.0",
    "com.google.http-client:google-http-client-gson:1.43.3",
    "com.google.http-client:google-http-client:1.43.3",
    "com.google.oauth-client:google-oauth-client:1.34.1",
    "com.google.protobuf:protobuf-java:3.24.3",

[IvanO-20]

+1, the same
error: emptyList() is not public in LazyStringArrayList; cannot be accessed from outside package com.google.protobuf.LazyStringArrayList.emptyList();
with 3.24.3 version

[googleberg]

The only way I can see to get this error is if you try to use newer gencode (that calls LazyStringArrayList.emptyList()) against an older runtime (where LazyStringArrayList.emptyList() is still package-private). This is not a supported configuration. Gencode from a particular release of protobuf will only work with the same runtime version or a later runtime in the same major version.

[RCRedWu]

why close this issue, it is still have an error!!!

[googleberg]

@RCRedWu No, this is working as intended. You cannot use gencode from a later version of protoc with an older runtime. If you do, the behavior is undefined and likely broken. We are adding explicit checks to prevent unintentional mixing of this type so that you get clear error messages.

[snazy]

@googleberg I think, the above requirement needs to be weakened. The current situation with protobuf v4 effectively makes upgrading impossible for a ton of projects. protobuf needs to provide a proper migration path for downstream projects as described in JLBP-7. See grpc/grpc-java#11015 and #16452

[googleberg]

@snazy We're unlikely to weaken the requirement that "new gencode will only work with matching runtime or later". We are reviewing a couple of proposals for how to ease the transition from 3.x to 4.x -- which would require relaxing the policy that "gencode and runtimes from different major versions are never compatible".