TLS registry named config and Quarkus Messaging Kafka or Kafka Client can lead to failed application startup

[michalvavrik]

Describe the bug

I have app using Quarkus Messaging Kafka, TLS registry and (one of the ways to reproduce it also uses Kafka streams other uses Kafka Client). Most of the time application fails to start.

Expected behavior

Application starts.

Actual behavior

Startup fails and following exception is logged:

2024-09-07 13:21:05,927 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:116)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:541)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:492)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:137)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer.<init>(KafkaStreamsProducer.java:107)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.doCreate(Unknown Source)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.create(Unknown Source)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer_Observer_onStartup_eonSDFgmJKlmEvopYKyyKONG1pI.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:351)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:333)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:80)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:156)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:107)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	... 11 more
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:252)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:189)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:525)
	... 32 more
Caused by: org.apache.kafka.common.KafkaException: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
	... 34 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at io.quarkus.tls.runtime.CertificateRecorder.get(CertificateRecorder.java:134)
	at io.quarkus.kafka.client.tls.QuarkusKafkaSslEngineFactory.configure(QuarkusKafkaSslEngineFactory.java:101)
	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180)
	... 38 more

And in case of KafkaAdminClient the stacktrace is:

16:42:45,524 INFO  [app] 16:42:43,577 Failed to start application: java.lang.RuntimeException: Failed to start quarkus
16:42:45,524 INFO  [app] 	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
16:42:45,524 INFO  [app] 	at io.quarkus.runtime.Application.start(Application.java:101)
16:42:45,524 INFO  [app] 	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
16:42:45,524 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
16:42:45,524 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
16:42:45,524 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
16:42:45,524 INFO  [app] 	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
16:42:45,524 INFO  [app] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
16:42:45,525 INFO  [app] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
16:42:45,525 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
16:42:45,525 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
16:42:45,525 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:264)
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.internals.ConsumerDelegateCreator.create(ConsumerDelegateCreator.java:65)
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:600)
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:595)
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:576)
16:42:45,525 INFO  [app] 	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:556)
16:42:45,525 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider.getSslConsumer(SslKafkaProvider.java:53)
16:42:45,525 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider_ProducerMethod_getSslConsumer_NFS8yLBckt8VZAzW5baLLACHtyY_Bean.doCreate(Unknown Source)
16:42:45,525 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider_ProducerMethod_getSslConsumer_NFS8yLBckt8VZAzW5baLLACHtyY_Bean.create(Unknown Source)
16:42:45,525 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider_ProducerMethod_getSslConsumer_NFS8yLBckt8VZAzW5baLLACHtyY_Bean.create(Unknown Source)
16:42:45,525 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
16:42:45,525 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
16:42:45,526 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider_ProducerMethod_getSslConsumer_NFS8yLBckt8VZAzW5baLLACHtyY_Bean.get(Unknown Source)
16:42:45,526 INFO  [app] 	at io.quarkus.qe.messaging.ssl.providers.SslKafkaProvider_ProducerMethod_getSslConsumer_NFS8yLBckt8VZAzW5baLLACHtyY_Bean.get(Unknown Source)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
16:42:45,526 INFO  [app] 	at io.quarkus.qe.messaging.ssl.quickstart.SslKafkaEndpoint.initialize(SslKafkaEndpoint.java:46)
16:42:45,526 INFO  [app] 	at io.quarkus.qe.messaging.ssl.quickstart.SslKafkaEndpoint_Observer_initialize_fXpzbA0Aag2X1uiiwypy52BybQ4.notify(Unknown Source)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:351)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:333)
16:42:45,526 INFO  [app] 	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:80)
16:42:45,527 INFO  [app] 	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:156)
16:42:45,527 INFO  [app] 	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:107)
16:42:45,527 INFO  [app] 	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
16:42:45,527 INFO  [app] 	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
16:42:45,527 INFO  [app] 	... 11 more
16:42:45,527 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
16:42:45,527 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:252)
16:42:45,527 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:162)
16:42:45,527 INFO  [app] 	at org.apache.kafka.clients.consumer.internals.ConsumerUtils.createConsumerNetworkClient(ConsumerUtils.java:86)
16:42:45,527 INFO  [app] 	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:191)
16:42:45,527 INFO  [app] 	... 41 more
16:42:45,527 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
16:42:45,527 INFO  [app] 	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:77)
16:42:45,528 INFO  [app] 	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
16:42:45,528 INFO  [app] 	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
16:42:45,528 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
16:42:45,528 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
16:42:45,528 INFO  [app] 	... 44 more
16:42:45,528 INFO  [app] Caused by: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
16:42:45,528 INFO  [app] 	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
16:42:45,528 INFO  [app] 	at io.quarkus.tls.runtime.CertificateRecorder.get(CertificateRecorder.java:134)
16:42:45,528 INFO  [app] 	at io.quarkus.kafka.client.tls.QuarkusKafkaSslEngineFactory.configure(QuarkusKafkaSslEngineFactory.java:101)
16:42:45,528 INFO  [app] 	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
16:42:45,528 INFO  [app] 	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
16:42:45,528 INFO  [app] 	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
16:42:45,529 INFO  [app] 	... 48 more

How to Reproduce?

I have 3 reproducers, in the first one, application fails to start always. In the second one (DEV mode) I experience it most of the time, but not always. You need to retry. However the first reproducer is build and started by QE FW and it might be less convenient for debugging. The third one follows Admin client docs.

First reproducer (JVM mode):

git clone [email protected]:michalvavrik/quarkus-test-suite.git -b feature/tls-registry-kafka
cd quarkus-test-suite/messaging/kafka-streams-reactive-messaging
mvn clean verify -Dit.test=SaslSslAlertMonitorIT

Second reproducer (DEV mode) (please note that Topology is not good, I know but it fails before it's validated anyway):

quarkus create app --extensions=messaging-kafka,kafka-streams,resteasy-jsonb,tls-registry

cd code-with-quarkus/
quarkus tls generate-certificate -n clement -p top-secret

# important: now update quarkus.tls.sasl-monitor.trust-store.p12.path with console output, e.g. Trust Store File: /tmp/code-with-quarkus/.certs/clement-truststore.p12 so take the truststore path

cat <<EOF >> ./src/main/resources/application.properties
quarkus.kafka-streams.topics=words
kafka.tls-configuration-name=sasl-monitor
quarkus.tls.sasl-monitor.trust-store.p12.password=top-secret
quarkus.tls.sasl-monitor.trust-store.p12.path=/tmp/strimzi-server-ssl-certs10979500127798639243/strimzi-server-ssl-truststore.p12
kafka.ssl.enable=true
ssl.endpoint.identification.algorithm=https
kafka.sasl.mechanism=SCRAM-SHA-512
kafka.security.protocol=SASL_SSL
kafka.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="client" password="client-secret12345678912345678912";
EOF

cat <<EOF > ./src/main/java/org/acme/TopologyProducer.java
package org.acme;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.inject.Produces;

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.kstream.GlobalKTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
import org.apache.kafka.streams.state.Stores;

import io.quarkus.kafka.client.serialization.ObjectMapperSerde;

@ApplicationScoped
public class TopologyProducer {

    static final String WEATHER_STATIONS_STORE = "weather-stations-store";

    private static final String WEATHER_STATIONS_TOPIC = "weather-stations";
    private static final String TEMPERATURE_VALUES_TOPIC = "temperature-values";
    private static final String TEMPERATURES_AGGREGATED_TOPIC = "temperatures-aggregated";

    @Produces
    public Topology buildTopology() {
        StreamsBuilder builder = new StreamsBuilder();

        ObjectMapperSerde<String> weatherStationSerde = new ObjectMapperSerde<>(
                String.class);
        ObjectMapperSerde<String> aggregationSerde = new ObjectMapperSerde<>(String.class);

        KeyValueBytesStoreSupplier storeSupplier = Stores.persistentKeyValueStore(
                WEATHER_STATIONS_STORE);

        GlobalKTable<String, String> stations = builder.globalTable(
                WEATHER_STATIONS_TOPIC,
                Consumed.with(Serdes.String(), weatherStationSerde));

        builder.stream(
                        TEMPERATURE_VALUES_TOPIC,
                        Consumed.with(Serdes.String(), Serdes.String())
                )
                .join(                                                        // <3>
                        stations,
                        (stationId, timestampAndValue) -> stationId,
                        (timestampAndValue, station) -> new String(station)
                )
                .groupByKey()                                                 // <4>
                .aggregate(                                                   // <5>
                        String::new,
                        (stationId, value, aggregation) -> aggregation,
                        Materialized.<String, String> as(storeSupplier)
                                .withKeySerde(Serdes.String())
                                .withValueSerde(aggregationSerde)
                )
                .toStream()
                .to(
                        TEMPERATURES_AGGREGATED_TOPIC
                );

        return builder.build();
    }
}
EOF

quarkus dev

Third reproducer - this one uses Kafka Client and Quarkus Messaging Kafka

What is does is that it implements Kafka reference example here https://quarkus.io/guides/kafka#kafka-bare-clients and it creates KafkaAdminClient like with properties:

@Inject
    @Identifier("default-kafka-broker")
    Map<String, Object> config;

git clone [email protected]:michalvavrik/quarkus-test-suite.git -b feature/kafka-tls-registry
cd quarkus-test-suite/messaging/kafkaSSL
mvn clean verify -Dit.test=KafkaSslWithTlsRegistryIT

Output of uname -a or ver

Fedora 40

Output of java -version

OpenJDK 21

Quarkus version or git rev

999-SNAPSHOT & 3.14.x

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.9

Additional information

No response

[quarkus-bot]

/cc @alesj (kafka), @cescoffier (kafka), @ozangunalp (kafka)

[michalvavrik]

@cescoffier I wrote race because it doesn't happen when Kafka Streams are not present and once I started the DEV mode and it didn't fail. I might have done mistake in that particular attempt, in which case it fails always.

[michalvavrik]

Ah, yeah, sometimes it takes longer for it to fail, now I saw it again.

[michalvavrik]

Oh, I just found 3rd way how to reproduce it, and this time it's just quarkus-kafka-client and quarkus-messaging-kafka, I'll update issue description with details in a bit.

[cescoffier]

Kafka streams is not really supported (let's say it's behind). If you have a reproducer with the reactive messaging connector it would be great.

[michalvavrik]

Kafka streams is not really supported (let's say it's behind). If you have a reproducer with the reactive messaging connector it would be great.

I have application with Quarkus Messaging Kafka and one of Kafka client/Kafka Streams. And if Quarkus sets io.quarkus.kafka.client.tls.QuarkusKafkaSslEngineFactory and it results in NPE, I'd say it is user experience issue.

I provided 3 different reproducers, if they are not supported, I suggest to improve validation error so that people like me don't spend time on this. Thank you

[gsmet]

AFAICS, there is a reproducer with just Kafka Client and Quarkus Messaging Kafka so we probably need to get things working.

Also, we shouldn't get a race condition and a NPE, even if the extension is not supported.

[cescoffier]

I agree that we need to fix it, but we will start with the connector. Kafka streams if the error is different will done in a second step.

[michalvavrik]

I agree that we need to fix it, but we will start with the connector. Kafka streams if the error is different will done in a second step.

Understand. As far as standalone reproducer, I don't have it ATM and I have had a long week. Next week if I am assigned to this I'll try to create one, but well, it's unlikely I'll. Please use the reproducers I have and if there is another bug, we need to wait for a user that reports it. Thank you

[cescoffier]

Well, seems that in all the reproducer it's the unmanaged client what is failing (admin, or the streams one). It is likely to not support the TLS registry. Now, it should not fail.

[michalvavrik]

Well, seems that in all the reproducer it's the unmanaged client what is failing (admin, or the streams one). It is likely to not support the TLS registry. Now, it should not fail.

I followed Quarkus Kafka reference to create that unmanaged client. I think it needs better validation and documentation then. Thanks

[michalvavrik]

I respect that they are not supported scenarios now that I know it. And you agree it needs to be documented, I am happy, thank you.

[michalvavrik]

@cescoffier I spend some more time on this and managed to reproduce this with managed Quarkus KafkaAdminClient.

19:36:07,314 INFO  [app] 19:36:05,471 Failed to start application: java.lang.RuntimeException: Failed to start quarkus
19:36:07,314 INFO  [app] 	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
19:36:07,314 INFO  [app] 	at io.quarkus.runtime.Application.start(Application.java:101)
19:36:07,315 INFO  [app] 	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
19:36:07,315 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
19:36:07,315 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
19:36:07,316 INFO  [app] 	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
19:36:07,316 INFO  [app] 	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
19:36:07,316 INFO  [app] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
19:36:07,316 INFO  [app] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
19:36:07,317 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
19:36:07,317 INFO  [app] 	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
19:36:07,317 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
19:36:07,317 INFO  [app] 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:541)
19:36:07,318 INFO  [app] 	at org.apache.kafka.clients.admin.Admin.create(Admin.java:147)
19:36:07,318 INFO  [app] 	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
19:36:07,318 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient.init(KafkaAdminClient.java:40)
19:36:07,318 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient_Bean.doCreate(Unknown Source)
19:36:07,319 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient_Bean.create(Unknown Source)
19:36:07,319 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient_Bean.create(Unknown Source)
19:36:07,319 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
19:36:07,319 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
19:36:07,320 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
19:36:07,320 INFO  [app] 	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c43(Unknown Source)
19:36:07,320 INFO  [app] 	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
19:36:07,320 INFO  [app] 	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
19:36:07,321 INFO  [app] 	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
19:36:07,321 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient_ClientProxy.arc$delegate(Unknown Source)
19:36:07,321 INFO  [app] 	at io.quarkus.kafka.client.runtime.KafkaAdminClient_ClientProxy.getTopics(Unknown Source)
19:36:07,321 INFO  [app] 	at io.quarkus.qe.messaging.ssl.quickstart.SslKafkaEndpoint.initialize(SslKafkaEndpoint.java:46)
19:36:07,322 INFO  [app] 	at io.quarkus.qe.messaging.ssl.quickstart.SslKafkaEndpoint_Observer_initialize_lpn1Kf5ZWJHu7ZdC9-r650-ixTY.notify(Unknown Source)
19:36:07,322 INFO  [app] 	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:351)
19:36:07,322 INFO  [app] 	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:333)
19:36:07,323 INFO  [app] 	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:80)
19:36:07,323 INFO  [app] 	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:156)
19:36:07,323 INFO  [app] 	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:107)
19:36:07,323 INFO  [app] 	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
19:36:07,324 INFO  [app] 	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
19:36:07,324 INFO  [app] 	... 11 more
19:36:07,324 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
19:36:07,325 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:252)
19:36:07,325 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:189)
19:36:07,325 INFO  [app] 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:525)
19:36:07,326 INFO  [app] 	... 35 more
19:36:07,326 INFO  [app] Caused by: org.apache.kafka.common.KafkaException: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
19:36:07,326 INFO  [app] 	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:77)
19:36:07,326 INFO  [app] 	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
19:36:07,327 INFO  [app] 	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
19:36:07,327 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
19:36:07,327 INFO  [app] 	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
19:36:07,327 INFO  [app] 	... 37 more
19:36:07,327 INFO  [app] Caused by: java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
19:36:07,327 INFO  [app] 	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
19:36:07,327 INFO  [app] 	at io.quarkus.tls.runtime.CertificateRecorder.get(CertificateRecorder.java:134)
19:36:07,327 INFO  [app] 	at io.quarkus.kafka.client.tls.QuarkusKafkaSslEngineFactory.configure(QuarkusKafkaSslEngineFactory.java:101)
19:36:07,327 INFO  [app] 	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
19:36:07,327 INFO  [app] 	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
19:36:07,328 INFO  [app] 	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
19:36:07,328 INFO  [app] 	... 41 more

steps:

git clone [email protected]:michalvavrik/quarkus-test-suite.git -b repro/managedkafka-client
cd quarkus-test-suite/messaging/kafkaSSL
mvn clean verify -Dit.test=KafkaSslWithTlsRegistryIT

[cescoffier]

I see 2 potential issues:

• the registry is not yet initialized. Normally we do that in the processor to make sure the registry is initialized, it may have been missed (what is odd is that it should have been an Arc issue)

• if cannot find the tls-configuration-name. Anyway if not found it should skip the whole thing.

[ozangunalp]

Agree on the second point.

We could've been slightly more defensive and check for tls-configuration-name key existing on the Kafka config map, but that custom SSL engine factory is used only when there is a tls-configuration-name set.

So maybe the registry is initialized but tls-configuration-name is not found. I need to debug to see why this is happening .

[ozangunalp]

Of course, internal admin clients (created for health checks and the Kafka streams extension) filter out non-admin-client property names to avoid the annoying Kafka log of unrecognized property names.
But I guess we don't have a choice in that case, we need to allow passing the tls-configuration-name key to the client configs.