Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KafkaStreamsITCase is flaky causing endless TimeoutException warnings leading to job being aborted #16415

Closed
famod opened this issue Apr 10, 2021 · 3 comments · Fixed by #16435
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/kafka area/kafka-streams kind/bug Something isn't working
Milestone

Comments

@famod
Copy link
Member

famod commented Apr 10, 2021

E.g. here and multiple times in #16249:

2021-04-09 21:29:04,547 INFO  [io.quarkus] (main) Profile prod activated. 
2021-04-09 21:29:04,547 INFO  [io.quarkus] (main) Installed features: [cdi, kafka-streams, micrometer, resteasy, resteasy-jackson, smallrye-health]
2021-04-09 21:29:14,537 ERROR [io.qua.kaf.str.run.KafkaStreamsProducer] (pool-4-thread-1) Failed to get topic names from broker: java.util.concurrent.TimeoutException
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer.waitForTopicsToBeCreated(KafkaStreamsProducer.java:306)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer.access$000(KafkaStreamsProducer.java:60)
	at io.quarkus.kafka.streams.runtime.KafkaStreamsProducer$1.run(KafkaStreamsProducer.java:172)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

The same can also happen in KafkaSnappyProducerITCase, e.g. here.

There seems to be a configuration issue, because before those timeouts the log is flooded with warnings like these (just en excerpt):

[INFO] Running io.quarkus.it.kafka.streams.KafkaStreamsITCase
2021-04-09 22:08:31,378 WARN  [io.qua.deployment] (main) Producing values from constructors and fields is no longer supported and will be removed in a future release: io.quarkus.deployment.annotations.BuildProducer io.quarkus.it.classtransformer.ClassTransformerProcessor.transformers
Executing [/home/runner/work/quarkus/quarkus/integration-tests/kafka-streams/target/quarkus-integration-test-kafka-streams-999-SNAPSHOT-runner, -Dquarkus.http.port=8081, -Dquarkus.http.ssl-port=8444, -Dtest.url=http://localhost:8081, -Dquarkus.log.file.path=/home/runner/work/quarkus/quarkus/integration-tests/kafka-streams/target/quarkus.log, -Dquarkus.log.file.enable=true]
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-04-09 22:08:38,753 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.location' was supplied but isn't a known config.
2021-04-09 22:08:38,779 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.login.refresh.buffer.seconds' was supplied but isn't a known config.
2021-04-09 22:08:38,779 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.password' was supplied but isn't a known config.
2021-04-09 22:08:38,779 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.truststore.type' was supplied but isn't a known config.
2021-04-09 22:08:38,779 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'ssl.endpoint.identification.algorithm' was supplied but isn't a known config.
2021-04-09 22:08:38,779 WARN  [org.apa.kaf.cli.adm.AdminClientConfig] (main) The configuration 'sasl.kerberos.ticket.renew.jitter' was supplied but isn't a known config.
2021-04-09 22:08:38,789 WARN  [org.apa.kaf.str.pro.int.StateDirectory] (main) Using /tmp directory in the state.dir property can cause failures with writing the checkpoint file due to the fact that this directory can be cleared by the OS
@famod famod added kind/bug Something isn't working area/housekeeping Issue type for generalized tasks not related to bugs or enhancements labels Apr 10, 2021
@quarkus-bot
Copy link

quarkus-bot bot commented Apr 10, 2021

/cc @cescoffier, @gunnarmorling, @rquinio

@cescoffier
Copy link
Member

Hopefully fixed by #16435

@famod famod linked a pull request Apr 14, 2021 that will close this issue
@famod
Copy link
Member Author

famod commented Apr 14, 2021

Thanks @cescoffier! I'll close this one. In case the problem pops up again we can always reopen this issue.

@famod famod closed this as completed Apr 14, 2021
@famod famod added this to the 2.0 - main milestone Apr 14, 2021
@gsmet gsmet modified the milestones: 2.0.0.Alpha1, 1.11.7.Final Apr 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/kafka area/kafka-streams kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants