Kafka SASL GSSAPI exception #20088

irenakezic · 2021-09-13T09:06:33Z

Describe the bug

Connecting to a Kafka broker using SASL GSSAPI works fine without the quarkus-infinispan-client extension. However, when quarkus-infinispan-client extension is added as a dependency the exception occurs when connecting to Kafka.

Expected behavior

Kafka connection is sucessful.

Actual behavior

The following exception occurs:

Caused by: javax.security.sasl.SaslException: ELY05108: Unable to create response token [Caused by javax.security.sasl.SaslException: ELY05127: No security layer supported by server but maximum message size received: "65536"]
	at org.wildfly.security.sasl.gssapi.GssapiClient.evaluateMessage(GssapiClient.java:313)
	at org.wildfly.security.sasl.util.AbstractSaslParticipant.evaluateMessage(AbstractSaslParticipant.java:219)
	at org.wildfly.security.sasl.gssapi.GssapiClient.evaluateChallenge(GssapiClient.java:218)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:524)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:524)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:431)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:301)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:176)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:563)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1329)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1260)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.security.sasl.SaslException: ELY05127: No security layer supported by server but maximum message size received: "65536"
	at org.wildfly.security.sasl.gssapi.GssapiClient.evaluateMessage(GssapiClient.java:281)
	... 15 more
java.security.PrivilegedActionException: javax.security.sasl.SaslException: ELY05108: Unable to create response token [Caused by javax.security.sasl.SaslException: ELY05127: No security layer supported by server but maximum message size received: "65536"] occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

How to Reproduce?

Reproducer:
https://github.com/irenakezic/kafka-sasl.git

Steps to reproduce:

Build project using "mvn clean package -Djava.security.krb5.conf=src/test/resources/krb5.conf"
Build should be successful and tests should run with no errors
Uncomment infinispan client extension dependency in pom.xml (you can also uncomment InfinispanBean code and infinispan config in application.properties though uncommenting the dependency in pom.xml is enough)
Test should fail with exception stated above

Please note that Kafka testcontainer in the example provided runs on port 9092 and container will not start properly if port is already taken on your localhost.

The reproducer is created using version 2.2.2.Final as it is the latest one but I need a fix for 1.11.7.Final-redhat-00009 if possible.

Output of `uname -a` or `ver`

No response

Output of `java -version`

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

1.11.7.Final-redhat-00009

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

The reason is wildfly.elytron.sasl.gssapi dependency that comes along with quarkus-infinispan-client. It seems that elytron GSSAPI implementation has more restrictive checks than default one.
Noticed that wildfly uses "wildfly.sasl.relax-compliance" config property to relax security checks for elytron gssapi. I wonder if there an option to add something similar in Quarkus?

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2021-09-13T09:06:34Z

/cc @cescoffier, @ozangunalp

cescoffier · 2021-09-13T10:43:30Z

I guess we will have to relax some checks.

cescoffier · 2021-09-13T10:48:58Z

@sberyozkin any advice?

cescoffier · 2021-09-19T15:18:01Z

@sberyozkin by any chance, you have an idea where this "wildfly.sasl.relax-compliance" property is managed in wildfly? I would like to see what it is actually doing.

irenakezic · 2021-09-19T16:05:37Z

@cescoffier
I think you can find this here: https://github.com/wildfly-security/wildfly-elytron/blob/1.x/sasl/gssapi/src/main/java/org/wildfly/security/sasl/gssapi/GssapiClient.java

cescoffier · 2021-09-19T18:00:19Z

Thanks @irenakezic !

I'm seeing https://github.com/wildfly-security/wildfly-elytron/blob/021ae1a096f5e864bbe4fc153256e2a2871ea0d3/sasl/gssapi/src/main/java/org/wildfly/security/sasl/gssapi/AbstractGssapiMechanism.java#L76, so I'm wondering why the system property does not work in Quarkus.

cescoffier · 2021-09-27T07:21:33Z

The wildfly.sasl.relax-compliance property can be passed in the channel configuration or even as a global configuration, but as it contains a - it gets replaced by wildfly.sasl.relax.compliance. This is something we can fix in reactive messaging.

The property gets replaced by wildfly.sasl.relax.compliance by the default configuration mapping (in Quarkus), which means, it cannot be found.

cescoffier · 2021-09-27T07:40:02Z

Workaround 1: Add mp.messaging.connector.smallrye-kafka.wildfly.sasl.relax-compliance=true to the configuration
Workaround 2: Add:

mp.messaging.outgoing.out.wildfly.sasl.relax-compliance=true
mp.messaging.incoming.in.wildfly.sasl.relax-compliance=true

The question is whether we want to automatically detect that Elytron is on the classpath and inject the configuration automatically.

@Ladicek @ozangunalp WDYT?

Ladicek · 2021-09-27T08:17:03Z

I don't really have an opinion, but I guess adding this property unconditionally for the Kafka connector can't hurt? :-)

ozangunalp · 2021-09-27T08:55:10Z

We can add the wildfly.sasl.relax-compliance property on kafka-client extension instead of smallrye-reactive-messaging-kafka no?

cescoffier · 2021-09-27T08:57:56Z

Yes, it would be in the kafka-client extension. Most probably injected directly in the global config map.

irenakezic · 2021-09-27T14:28:50Z

Hi @cescoffier,
thx for the workaround!
It works fine for Quarkus 2.2.2.Final. However, for Quarkus 1.11.7.Final-redhat-00009 (this the one I need to use for now) I get the following exceptuion:
2021-09-27 16:12:55,886 WARN [org.apa.kaf.cli.NetworkClient] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Error connecting to node localhost:9092 (id: -1 rack: null): java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed] at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348) at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) at org.apache.kafka.common.network.Selector.connect(Selector.java:256) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1006) at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:75) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1177) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1065) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:228) at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338) ... 10 more Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator Caused by: java.lang.ClassCastException: class java.lang.Boolean cannot be cast to class java.lang.String (java.lang.Boolean and java.lang.String are in module java.base of loader 'bootstrap') at org.wildfly.security.sasl.gssapi.AbstractGssapiMechanism.<init>(AbstractGssapiMechanism.java:76) at org.wildfly.security.sasl.gssapi.GssapiClient.<init>(GssapiClient.java:66) at org.wildfly.security.sasl.gssapi.GssapiClientFactory.createSaslClient(GssapiClientFactory.java:42) at java.security.sasl/javax.security.sasl.Sasl.createSaslClient(Sasl.java:433) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslClient$0(SaslClientAuthenticator.java:217) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:213) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:204) at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:275) at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:216) at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:142) at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:224) at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338) at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) at org.apache.kafka.common.network.Selector.connect(Selector.java:256) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1006) at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:75) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1177) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1065) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240) at java.base/java.lang.Thread.run(Thread.java:834)

Do you have any suggestions on how to workaround this one?

cescoffier · 2021-09-27T15:04:41Z

Did you try both workarounds?

The problem is that it sees "true", and transforms it to a boolean in 1.11, but AbstractGssapiMechanism wants it as a String.

irenakezic · 2021-09-27T15:09:29Z

@cescoffier I did try both options :(

When you use Infinispan and Kafka at the same time and you connect to a Kafka broker with SASL, the SASL client uses Elytron. However, Elytron SASL implementation is stricter than what Kafka expects. This commit relaxes Elytron checks for Kafka.

cescoffier · 2021-09-27T15:16:23Z

Unfortunately, there is nothing we can do. How Elytron is configured is not really "Kafka-friendly" ("-" instead of ".", String value instead of object)...

irenakezic · 2021-09-27T15:21:18Z

@cescoffier thx, I guess we will need to migrate to a more recent version than.

When you use Infinispan and Kafka at the same time and you connect to a Kafka broker with SASL, the SASL client uses Elytron. However, Elytron SASL implementation is stricter than what Kafka expects. This commit relaxes Elytron checks for Kafka. (cherry picked from commit c17f875)

irenakezic added the kind/bug Something isn't working label Sep 13, 2021

quarkus-bot bot added the area/kafka label Sep 13, 2021

cescoffier self-assigned this Sep 26, 2021

cescoffier mentioned this issue Sep 27, 2021

Relax Elytron SASL check when used with Kafka #20416

Merged

cescoffier closed this as completed in #20416 Sep 28, 2021

quarkus-bot bot added this to the 2.4 - main milestone Sep 28, 2021

geoand modified the milestones: 2.4 - main, 2.3.0.Final Sep 28, 2021

jponge mentioned this issue Oct 6, 2021

Upgrade to Mutiny 1.1.0 and Mutiny Vert.x bindings 2.14.0 jponge/quarkus#57

Closed

ozangunalp mentioned this issue Dec 23, 2021

4 warnings about unknown 'wildfly.sasl.relax-compliance' config when using infinispan-client + smallrye-reactive-messaging-kafka #22495

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka SASL GSSAPI exception #20088

Kafka SASL GSSAPI exception #20088

irenakezic commented Sep 13, 2021

quarkus-bot bot commented Sep 13, 2021

cescoffier commented Sep 13, 2021

cescoffier commented Sep 13, 2021

cescoffier commented Sep 19, 2021

irenakezic commented Sep 19, 2021

cescoffier commented Sep 19, 2021

cescoffier commented Sep 27, 2021 •

edited

Loading

cescoffier commented Sep 27, 2021

Ladicek commented Sep 27, 2021

ozangunalp commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

Kafka SASL GSSAPI exception #20088

Kafka SASL GSSAPI exception #20088

Comments

irenakezic commented Sep 13, 2021

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Sep 13, 2021

cescoffier commented Sep 13, 2021

cescoffier commented Sep 13, 2021

cescoffier commented Sep 19, 2021

irenakezic commented Sep 19, 2021

cescoffier commented Sep 19, 2021

cescoffier commented Sep 27, 2021 • edited Loading

cescoffier commented Sep 27, 2021

Ladicek commented Sep 27, 2021

ozangunalp commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

cescoffier commented Sep 27, 2021

irenakezic commented Sep 27, 2021

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

cescoffier commented Sep 27, 2021 •

edited

Loading