Apicurio Kafka registry 2.1.5.Final doesn't work on Quarkus Upstream

[pjgg]

Describe the bug

QuarkusVersion: 999-SNAPSHOT
extension: quarkus-apicurio-registry-avro
Kafka version: quay.io/strimzi/kafka:0.28.0-kafka-3.1.0
apicurio: quay.io/apicurio/apicurio-registry-mem:2.1.5.Final

Description: When a Strimzi Kafka Quarkus app starts throw the following error:

14:13:16,172 SRMSG18228: A failure has been reported for Kafka topics '[stock-price]': io.apicurio.registry.rest.client.exception.RestClientException: RESTEASY003210: Could not find resource for full path: http://localhost:49263/apis/registry/v2/ids/globalIds/1/references

More details:

curl -v http://localhost:49263/apis/registry/v2/ids/globalIds/1/references

*   Trying 127.0.0.1:49263...
* Connected to localhost (127.0.0.1) port 49263 (#0)
> GET /apis/registry/v2/ids/globalIds/1/references HTTP/1.1
> Host: localhost:49263
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Thu, 26 May 2022 12:14:15 GMT
< Expires: Wed, 25 May 2022 12:14:15 GMT
< Pragma: no-cache
< Cache-control: no-cache, no-store, must-revalidate
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< Content-Type: application/json
< Content-Length: 331
< 
* Connection #0 to host localhost left intact
{"message":"RESTEASY003210: Could not find resource for full path: http://localhost:49263/apis/registry/v2/ids/globalIds/1/references","error_code":404,"detail":"NotFoundException: RESTEASY003210: Could not find resource for full path: http://localhost:49263/apis/registry/v2/ids/globalIds/1/references","name":"NotFoundException"}% 

Note that Apicurio was updated on Quarkus upstream to version 2.2.3.Final.
commitRef: a9ef280

If I update my Apicurio version to 2.2.3.Final then works as expected. Looks like this upgrade is not backward compatible with previous or already deployed Quarkus apps and forces you to upgrade Quarkus version in your apps and also Apicurio.

[quarkus-bot]

/cc @alesj, @cescoffier, @ozangunalp

[vsevel]

Hello,
I do have a similar issue:

io.apicurio.registry.rest.client.exception.RestClientException: RESTEASY003210: Could not find resource for full path: http://broker-schema-registry-infra-kafka-sbiz.apps.dev.ocp.dev.biz.lodh.com/apis/registry/v2/ids/globalIds/115/references
	at io.apicurio.registry.rest.client.impl.ErrorHandler.handleErrorResponse(ErrorHandler.java:64)
	at io.apicurio.rest.client.response.ResponseHandler.handle(ResponseHandler.java:39)
	at io.apicurio.rest.client.response.ResponseHandler.handle(ResponseHandler.java:19)
	at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:400)
...
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

I am using io.apicurio.apicurio-registry-serdes-avro-serde-2.2.4.Final.jar in a Quarkus 2.10.2.Final app.
And apicurio registry is 2.0.2.
I see that the ids/globalIds/x/references endpoint is only available in 2.2.x.
I take it the only option at this point is to upgrade to 2.2.3.Final for now?
and is this the plan for quarkus to support multiple older versions of the registry?
where is the issue? in quarkus, or in the apicurio client lib?
thanks.

[gsmet]

@ozangunalp @alesj could you have a look at this one?

We probably need to be extra cautious when upgrading if things are not backward compatible.

@cescoffier you might know who to ping on the Apicurio side to have some additional insights about how to best deal with it.

[cescoffier]

@EricWittmann @carlesarnal any insight on your side?

[vsevel]

I checked on the product side of red hat there are only 2.0.x versions of the registry.
so my options are basically not ideal:

• go to the non supported upstream apicurio project and upgrade to 2.2
• downgrade the client lib used in our quarkus apps (with the risk that client lib 2.0 might not be working with the newest versions of Quarkus)

[cescoffier]

we may need to switch back to 2.0. Waiting for @EricWittmann and @carlesarnal.

[vsevel]

I am not in a hurry @cescoffier . Quarkus is not part of the Kafka pilot apps that we push to prod before the end of the year. although I am trying to make sure Quarkus is fully supported internally on all use cases, and it is better communication if I can say that everything just works ;) . but I can wait.
I need to check what it means on the SB side.

[vsevel]

so I downgraded to 2.1.5, working with a 2.0 registry. and so far it seems to be working well.

[carlesarnal]

Hi all, sorry, I was on PTO. This is exactly the same problem discussed in #26322. The options are either to force the version of the client lib (to use a version compatible with the old server) or to upgrade the apicurio registry server version.

[cescoffier]

@ozangunalp @alesj WDYT?

Should be rollback and document the working sets?