Quarkus Vault problem

[RustamSultansoy]

Describe the bug

I have a small application with 2 rest endpoints and vault. Authentication method for vault is username/password
After 10-15 mins running app throws exception on API calls because it try to resolve "resteasy.original.webapplicationexception.behavior" property. And when it comes to vault source it couldn't authenticate second time. Thrown exception:
The current thread cannot be blocked: vert.x-eventloop-thread-8

Expected behavior

resteasy.original.webapplicationexception.behavior has default value. shouldn't be taken from vault.
vault also shouldn't throw this exception

Actual behavior

vault throws exception:
ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-8) HTTP Request to ... failed, error id: 180723b6-de12-4517-af3d-878cd0bb010c-1: java.lang.IllegalStateException: The current thread cannot be blocked: vert.x-eventloop-thread-8
at io.smallrye.mutiny.operators.UniBlockingAwait.await(UniBlockingAwait.java:29)
at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:61)
at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:472)
at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:457)
at io.quarkus.vault.runtime.client.VertxVaultClient.get(VertxVaultClient.java:438)
at io.quarkus.vault.runtime.client.VertxVaultClient.lookupSelf(VertxVaultClient.java:213)
at io.quarkus.vault.runtime.VaultAuthManager.validate(VaultAuthManager.java:107)
at io.quarkus.vault.runtime.VaultAuthManager.login(VaultAuthManager.java:89)
at io.quarkus.vault.runtime.VaultAuthManager.login(VaultAuthManager.java:78)
at io.quarkus.vault.runtime.VaultAuthManager.getClientToken(VaultAuthManager.java:62)
at io.quarkus.vault.runtime.VaultKvManager.readSecret(VaultKvManager.java:32)
at io.quarkus.vault.runtime.VaultKvManager_ClientProxy.readSecret(VaultKvManager_ClientProxy.zig:222)
at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:85)
at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:81)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:81)
at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$getSecretConfig$0(VaultConfigSource.java:65)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:65)
at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:51)
at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:26)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:27)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:27)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:27)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:22)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:84)
at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:65)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:26)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
at org.jboss.resteasy.microprofile.client.DefaultResponseExceptionMapper.handles(DefaultResponseExceptionMapper.java:27)
at org.jboss.resteasy.microprofile.client.ExceptionMapping.filter(ExceptionMapping.java:62)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterResponse(ClientInvocation.java:718)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:492)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:149)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
at com.sun.proxy.$Proxy77.readUserRoles(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.resteasy.microprofile.client.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:147)
at com.sun.proxy.$Proxy80.readUserRoles(Unknown Source)
at ch.skycell.security.keycloak.SkyMindKeycloakClient_0f71c4ae6073cff1d752acbd9d4726bd23713289_Synthetic_ClientProxy.readUserRoles(SkyMindKeycloakClient_0f71c4ae6073cff1d752acbd9d4726bd23713289_Synthetic_ClientProxy.zig:131)
at ch.skycell.security.rolesaugmentor.RolesAugmentor.build(RolesAugmentor.java:41)
at ch.skycell.security.rolesaugmentor.RolesAugmentor.augment(RolesAugmentor.java:32)
at ch.skycell.security.rolesaugmentor.RolesAugmentor_ClientProxy.augment(RolesAugmentor_ClientProxy.zig:189)
at io.quarkus.security.runtime.QuarkusIdentityProviderManagerImpl.handleIdentityFromProvider(QuarkusIdentityProviderManagerImpl.java:167)
at io.quarkus.security.runtime.QuarkusIdentityProviderManagerImpl.access$500(QuarkusIdentityProviderManagerImpl.java:30)
at io.quarkus.security.runtime.QuarkusIdentityProviderManagerImpl$2.apply(QuarkusIdentityProviderManagerImpl.java:112)
at io.quarkus.security.runtime.QuarkusIdentityProviderManagerImpl$2.apply(QuarkusIdentityProviderManagerImpl.java:109)
at io.smallrye.mutiny.operators.UniOnItemTransformToUni.invokeAndSubstitute(UniOnItemTransformToUni.java:31)
at io.smallrye.mutiny.operators.UniOnItemTransformToUni$2.onItem(UniOnItemTransformToUni.java:74)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.onItem(UniSerializedSubscriber.java:85)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.lambda$onItem$1(ContextPropagationUniInterceptor.java:31)
at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$1.onItem(ContextPropagationUniInterceptor.java:31)
at io.smallrye.mutiny.operators.uni.builders.DefaultUniEmitter.complete(DefaultUniEmitter.java:36)
at io.quarkus.smallrye.jwt.runtime.auth.MpJwtValidator$1.accept(MpJwtValidator.java:54)
at io.quarkus.smallrye.jwt.runtime.auth.MpJwtValidator$1.accept(MpJwtValidator.java:49)
at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribing(UniCreateWithEmitter.java:24)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:49)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:31)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.lambda$subscribing$0(ContextPropagationUniInterceptor.java:47)
at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.subscribing(ContextPropagationUniInterceptor.java:47)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:58)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:52)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:31)
at io.smallrye.mutiny.operators.UniOnItemTransformToUni.subscribing(UniOnItemTransformToUni.java:65)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:49)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:31)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.lambda$subscribing$0(ContextPropagationUniInterceptor.java:47)
at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.subscribing(ContextPropagationUniInterceptor.java:47)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:58)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:52)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:31)
at io.smallrye.mutiny.operators.UniMemoizeOp.subscribing(UniMemoizeOp.java:70)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:49)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:31)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.lambda$subscribing$0(ContextPropagationUniInterceptor.java:47)
at io.smallrye.context.impl.wrappers.SlowContextualExecutor.execute(SlowContextualExecutor.java:19)
at io.smallrye.mutiny.context.ContextPropagationUniInterceptor$2.subscribing(ContextPropagationUniInterceptor.java:47)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:58)
at io.smallrye.mutiny.operators.UniSerializedSubscriber.subscribe(UniSerializedSubscriber.java:52)
at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:50)
at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:104)
at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:51)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:137)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
at io.quarkus.vertx.http.runtime.cors.CORSFilter.handle(CORSFilter.java:92)
at io.quarkus.vertx.http.runtime.cors.CORSFilter.handle(CORSFilter.java:18)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:137)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
at io.quarkus.vertx.http.runtime.devmode.VertxHttpHotReplacementSetup$3.handle(VertxHttpHotReplacementSetup.java:86)
at io.quarkus.vertx.http.runtime.devmode.VertxHttpHotReplacementSetup$3.handle(VertxHttpHotReplacementSetup.java:75)
at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)

Configuration

quarkus.vault.url=http://url
quarkus.vault.authentication.userpass.username=user
quarkus.vault.authentication.userpass.password=pass
quarkus.vault.kv-secret-engine-mount-path=space
quarkus.vault.secret-config-kv-path=app

Quarkus 1.12.2

[quarkus-bot]

/cc @vsevel

[RustamSultansoy]

fyi. i added manually resteasy.original.webapplicationexception.behavior=false to properties file, but still get same exception, because it try to find this property for given profile like "%dev.resteasy.original.webapplicationexception.behavior"

[RustamSultansoy]

this properties helped
quarkus.vault.secret-config-cache-period=1500H
resteasy.original.webapplicationexception.behavior=false

[vsevel]

what happens is that you read (or get injected) a property during the execution of the program. as a config source, vault gets called to attempt retrieving the particular property you are mentioning.
properties fetched from vault get cached in the vault config source (by default 10 mins). this means that after 10 mins of the initial retrieval, we will try to refresh those properties coming from vault.
recently we changed the underlying client from okhttp to vertx. in the vault client we are making blocking calls through request.send().await().atMost(getRequestTimeout()). but when this happens in a vert.x-eventloop-thread, an error gets raised.
setting quarkus.vault.secret-config-cache-period to a big value deactivates the cache in the vault config source effectively. we need to better handle this.
@cescoffier what is the proper way to approach this? should we just deactivate the cache refresh when we see that we are in the a vert.x-eventloop-thread (ignoring the exception when in VaultConfigSource.fetchSecrets or looking at the name of the thread, and ignore the attempt to refresh the cache)? WDYT?

[cescoffier]

I suppose that the method is synchronous (not returning an uni), in this case, yes, unfortunately, you will need to check if you can block, and if you can, use the await(), otherwise probably print a warning and keep the cached value.