Kafka OAUTH Keycloak integration test failing with GraalVM / Mandrel 22.1-dev

[zakkak]

Describe the bug

$title

Expected behavior

Test should pass

Actual behavior

Test fails with:

2022-02-03 04:06:48,325 ERROR [org.apa.kaf.cli.NetworkClient] (smallrye-kafka-consumer-thread-0) [Consumer clientId=kafka-consumer-in, groupId=quarkus-integration-test-kafka-oauth-keycloak] Connection to node -1 (localhost/127.0.0.1:9092) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: No OAuth *** in Subject's private credentials [Caused by java.io.IOException: No OAuth *** in Subject's private credentials]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
2022-02-03 04:06:48,325 WARN  [org.apa.kaf.cli.NetworkClient] (smallrye-kafka-consumer-thread-0) [Consumer clientId=kafka-consumer-in, groupId=quarkus-integration-test-kafka-oauth-keycloak] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
2022-02-03 04:06:48,339 ERROR [org.apa.kaf.cli.NetworkClient] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Connection to node -1 (localhost/127.0.0.1:9092) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: No OAuth *** in Subject's private credentials [Caused by java.io.IOException: No OAuth *** in Subject's private credentials]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
2022-02-03 04:06:48,339 WARN  [org.apa.kaf.cli.NetworkClient] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
2022-02-03 04:06:48,340 ERROR [io.sma.rea.mes.kafka] (smallrye-kafka-producer-thread-0) SRMSG18206: Unable to write to Kafka from channel out (topic: mytopic): org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: No OAuth *** in Subject's private credentials [Caused by java.io.IOException: No OAuth *** in Subject's private credentials]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: No OAuth *** in Subject's private credentials [Caused by java.io.IOException: No OAuth *** in Subject's private credentials]
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:120)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
	at java.security.AccessController.executePrivileged(AccessController.java:145)
	at java.security.AccessController.doPrivileged(AccessController.java:106)
	at javax.security.auth.Subject.doAs(Subject.java:36)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
	at java.lang.Thread.run(Thread.java:829)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:688)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:202)
Caused by: java.io.IOException: No OAuth *** in Subject's private credentials
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:104)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	... 17 more

See https://github.com/graalvm/mandrel/runs/5046248182?check_suite_focus=true#step:11:1932

How to Reproduce?

No response

Output of uname -a or ver

GH runner

Output of java -version

11.0.14 and 17.0.2

GraalVM version (if different from Java)

22.1.0-dev51ebdca

Quarkus version or git rev

14f086d

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Karm, @cescoffier, @galderz, @ozangunalp, @pedroigor, @sberyozkin

[sberyozkin]

@ozangunalp, Hi, FYI, I ended up using Keycloak-X in the OIDC MTLS test: https://github.com/quarkusio/quarkus/blob/main/integration-tests/oidc/src/test/java/io/quarkus/it/keycloak/KeycloakXTestResourceLifecycleManager.java

[cescoffier]

That's an annoying one. We cannot really change the Kafka internal classes.

@sberyozkin Do you believe it's because of the switch to keycloak-x?

[sberyozkin]

Hey @cescoffier

No, not at all; I meant, for the tests, instead of using WildFly based distro as is done in the Kafka test, you can use a Keycloak-X container instead, it should not make a difference for the actual Kafka code. When I was preparing an OIDC MTLS test I actually copied this test code first :-), but found it was too difficult to setup MTLS so found it going much easier with KC-X.
However, it does not support the realm import from a file at the startup, can only be done via API.

It might make sense to tweak Dev Services for Keycloak (KC-X is the default image) to support the key stores and truststores, you'd then be able to reuse it with Kafka. May be a longer term enhancement

[ozangunalp]

@sberyozkin thanks for the heads up.
I still don't get why this test fails on Mandrel 22.1-dev

[cescoffier]

Yeah, not sure how we can make progress on this one (while we need to fix it).

@zakkak where can I grab a Mandrel 22.1-dev package?

[zakkak]

@cescoffier dev builds are available at https://github.com/graalvm/graalvm-ce-dev-builds/releases/

I will try to have another look at it again this week, last time I tried I couldn't get the test to pass even with 22.0 so please let me know if there are any special steps I need to take in order for it to run.

I am getting:

2022-02-14 09:24:50,590 ERROR [🐳 .io/.0.2]] (pool-3-thread-1) Could not start container: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*WFLYSRV0025.*'
	at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:49)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
	at io.quarkus.it.kafka.KafkaKeycloakTestResource.start(KafkaKeycloakTestResource.java:28)
	at io.quarkus.test.common.TestResourceManager$TestResourceEntryRunnable.run(TestResourceManager.java:452)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

I use

./mvnw -Dnative -pl integration-tests/kafka-oauth-keycloak -Dtest-containers -Dstart-containers -Dnative.surefire.skip -Dformat.skip -Dno-descriptor-tests clean verify

to run the test.

I even tried increasing the time out for the check but it didn't seem to work.

[cescoffier]

Hum. @sberyozkin see above... Sounds like it cannot start Keycloak.

[ozangunalp]

I started to migrate that test to use KeycloakX, will update this issue with my results.

[sberyozkin]

@ozangunalp Great, thanks

[galderz]

@sberyozkin @ozangunalp See zulip...

[cescoffier]

I would not migrate all our test to KaycloakX. We still have users on "classic" keycloak, and if we start seeing behavior differences, it can be tricky. I don't mind migrating these, but we need to keep that in mind.