Upgrading from 1.11.x to 1.13.x breaks kubernetes authentication for quarkus-vault extension

[fmarissel]

Describe the bug

I use quarkus-vault extension to access secrets dynamically.

I use client-token authentication for local development and kubernetes authentication for production (see below for configuration).

I upgraded from version 1.11.x to 1.13.x and everything works fine for local development. But when I tried to deploy into kubernetes, I get the following exception during kubernetes authentication :

ERROR: Failed to start application (with profile prod)
io.smallrye.mutiny.TimeoutException
	at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:58)
	at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:61)
	at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:161)
	at io.quarkus.vault.runtime.client.VertxVaultClient.exec(VertxVaultClient.java:154)
	at io.quarkus.vault.runtime.client.VertxVaultClient.post(VertxVaultClient.java:107)
	at io.quarkus.vault.runtime.client.VertxVaultClient.post(VertxVaultClient.java:101)
	at io.quarkus.vault.runtime.client.authmethod.VaultInternalKubernetesAuthMethod.login(VaultInternalKubernetesAuthMethod.java:28)
	at io.quarkus.vault.runtime.VaultAuthManager.loginKubernetes(VaultAuthManager.java:256)
	at io.quarkus.vault.runtime.VaultAuthManager.login(VaultAuthManager.java:155)
	at io.quarkus.vault.runtime.VaultAuthManager.vaultLogin(VaultAuthManager.java:145)
	at io.quarkus.vault.runtime.VaultAuthManager.login(VaultAuthManager.java:116)
	at io.quarkus.vault.runtime.VaultAuthManager.login(VaultAuthManager.java:95)
	at io.quarkus.vault.runtime.VaultAuthManager.getClientToken(VaultAuthManager.java:79)
	at io.quarkus.vault.runtime.VaultKvManager.readSecret(VaultKvManager.java:36)
	at io.quarkus.vault.runtime.VaultKvManager_ClientProxy.readSecret(VaultKvManager_ClientProxy.zig:222)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:92)
	at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:88)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:88)
	at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$getSecretConfig$0(VaultConfigSource.java:72)
	at java.base/java.util.Optional.ifPresent(Optional.java:176)
	at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:72)
	at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:53)
	at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
	at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:26)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:27)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfigSourceInterceptor.getValue(SmallRyeConfigSourceInterceptor.java:27)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:22)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:26)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.ProfileConfigSourceInterceptor.convertProfile(ProfileConfigSourceInterceptor.java:125)
	at io.smallrye.config.ProfileConfigSourceInterceptor.<init>(ProfileConfigSourceInterceptor.java:49)
	at io.smallrye.config.SmallRyeConfigBuilder$1.getInterceptor(SmallRyeConfigBuilder.java:165)
	at io.smallrye.config.SmallRyeConfigBuilder$InterceptorWithPriority.getInterceptor(SmallRyeConfigBuilder.java:413)
	at io.smallrye.config.SmallRyeConfig$ConfigSourceInterceptorWithPriority.getInterceptor(SmallRyeConfig.java:608)
	at io.smallrye.config.SmallRyeConfig$ConfigSources.<init>(SmallRyeConfig.java:424)
	at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:66)
	at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:358)
	at io.quarkus.runtime.generated.Config.readConfig(Config.zig:2071)
	at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(RuntimeConfigSetup.zig:60)
	at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:552)
	at io.quarkus.runtime.Application.start(Application.java:90)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:100)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:66)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:42)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:119)
	at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)

Configuration

# vault
"%dev":
  quarkus:
    vault:
      url: ${VAULT_URL}
      authentication:
        client-token: 00000000-0000-0000-0000-000000000000
      secret-config-kv-path: ${VAULT_PATH}

"%prod":
  quarkus:
    vault:
      url: ${VAULT_URL}
      authentication:
        kubernetes:
          role: ${VAULT_K8S_ROLE}
          auth-mount-path: ${K8S_AUTH_MOUNT_PATH}
      secret-config-kv-path: ${VAULT_PATH}

Environment (please complete the following information):

Docker image

openjdk:14

Output of java -version

Java 14

Build tool (ie. output of mvnw --version or gradlew --version)

Maven 3.6

Additional context

I tried one month ago to upgrade from 1.11.x to 1.12.x and had the same error.

I didn't change anything else except the following property after upgrade :

quarkus:
  ...
  package:
    type: legacy-jar

[quarkus-bot]

/cc @geoand, @vsevel

[vsevel]

hello @fmarissel, try setting a bigger read timeout. the default is 1 sec: quarkus.vault.read-timeout=5S

[fmarissel]

hello @vsevel, I increased the value (I also tried 30S) but i still got the same error :/

[sberyozkin]

It might be a configuration issue ? (vault.url is not substituted ?, etc). It can make sense to report the connection URL at a trace log level if the connection fails

[fmarissel]

After some research, it seems vault authentication is now handled by vertx web client. And vertx web client ignore JVM settings like NON_PROXY_HOSTS (for VAULT_URL) which is the cause of the timeout

[vsevel]

it seems you found out the root cause. separately I validated that running a kubernetes auth with a VAULT_URL env variable actually worked for me. so it is not an issue with property injection.
it sounds like we would need something like eclipse-vertx/vert.x#2600
/cc @cescoffier

[vsevel]

hi @fmarissel some work has been done on eclipse-vertx/vert.x#3919, which may solve the issue you are reporting.
One thing you could try is upgrade to Quarkus 2.0 alpha, and upgrade vertx as well.
/cc @vietj

[vsevel]

eclipse-vertx/vert.x#3919 has been merged.
once vertx 4.1 is available in quarkus, we will need to to expose the new nonProxyHost property in the vault runtime config, and use it in the client
/cc @vietj

[fmarissel]

hi @vsevel sorry i wasn't available last week. I tried to upgrade to 2.0.0.Alpha2 but quarkus-universe-bom is missing and i didn't see anything about that in the migation guide :
https://github.com/quarkusio/quarkus/wiki/Migration-Guide-2.0

[geoand]

I tried to upgrade to 2.0.0.Alpha2 but quarkus-universe-bom is missing

You can use quarkus-bom instead

[fmarissel]

I upgraded to Quarkus 2.0.0.Alpha2 and vertx-core 4.1.0.Beta1 but I don't find a way to pass nonProxyHost variable to vertx web client

[vsevel]

you would have to add the option to the vault extension in here and recompile a new vault extension.

here is the new option in vertx.

[gsmet]

@vsevel we merged the upgrade to Vert.x 4.1.0.Beta1 this week. Would it be good enough to prepare a PR for this?

[gsmet]

I checked and unfortunately this feature is not in 4.1.0.Beta1.

I created a PR for the enhancement here #17369 so that we keep track of it.

[cescoffier]

Yes, it will be in vertx 4.1.0.CR1

[vsevel]

@gsmet I was planning to do it as soon as it would be available.