Infinispan Client is ignoring configuration environment variables

[ricardozanini]

Describe the bug
Unless specified in application.properties file, configurations for Infinispan Client are being ignored when they are only available via environment variables.

For example, if I have an empty application.properties file and set QUARKUS_INFINISPAN_CLIENT_SERVER_LIST environment variable, the client fails to read it and do not connect:

2019-11-22 16:28:39,544 INFO  [org.inf.cli.hot.RemoteCacheManager] (main) ISPN004021: Infinispan version: 10.0.0.CR1
2019-11-22 16:28:39,589 ERROR [org.inf.cli.hot.imp.ope.RetryOnFailureOperation] (HotRod-client-async-pool-1-1) ISPN004007: Exception encountered. Retry 10 out of 10: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /127.0.0.1:11222
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused

(tries to connect to localhost, the default value)

If I set the property quarkus.infinispan-client.server-list in the properties file to an empty value, then the environment variable is read and the client successfully connects to the server.

Expected behavior

That the Infinispan client extension reads the environment variable without having to explicit set it into an application.properties file.

Actual behavior

The desired property is not being read from environment variable if it's not presented in the properties file.

To Reproduce
Steps to reproduce the behavior:

1. Clone this project:

$ git clone --single-branch --branch remove-props https://github.com/ricardozanini/kogito-examples
$ cd kogito-examples/onboarding-example/onboarding

2. Set the environment variable to something like export QUARKUS_INFINISPAN_CLIENT_SERVER_LIST=http://im-not-here:11222
3. Build it with mvn clean package -Ppersistence -DskipTests
4. Run it: java -jar target/onboarding-8.0.0-SNAPSHOT-runner.jar
5. Check the logs and you'll see the message described above (failing to connect to localhost)

Configuration
Blank properties

Environment (please complete the following information):

• Output of uname -a or ver: Linux skywalker 5.3.11-200.fc30.x86_64 Switch to the Maven distributed copy of the SubstrateVM annotations #1 SMP Tue Nov 12 19:25:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Output of java -version:
  openjdk version "1.8.0_232"
  OpenJDK Runtime Environment (build 1.8.0_232-20191008104205.buildslave.jdk8u-src-tar--b07)
  OpenJDK 64-Bit GraalVM CE 19.2.1 (build 25.232-b07-jvmci-19.2-b03, mixed mode)

• Quarkus version or git rev: 0.27.0

Additional context
Would be nice to have such feature working, so we could configure Quarkus application in containers much easier.

Might be related: #1887

[gsmet]

@dmlloyd is it related to all the config fixes that just went in?

[dmlloyd]

Yes, this is something that was fixed with #5387.

[ricardozanini]

Thanks @dmlloyd
I assume this will be fixed in 1.0.0 1.1.0?

[dmlloyd]

Correct, 1.1.0.

[geoand]

Is this still an issue?

[ricardozanini]

@geoand lol this is old, I think it was fixed.

[geoand]

Thanks!

[israel-ramos]

@dmlloyd @gsmet
This issue persists.
The ENV vars is ignored.
For instance: QUARKUS_INFINISPAN_CLIENT__NAMED_INFINISPAN_CLIENTS__HOSTS
The same configs on application.properties works.
Version: quarkusPlatformVersion=3.11.0.CR1

[dmlloyd]

Are you sure you're using the correct config key? What is the equivalent key in application.properties that you are trying to set?

/cc @karesti

[karesti]

Hey, QUARKUS_INFINISPAN_CLIENT__NAMED_INFINISPAN_CLIENTS__HOSTS it's not a valid key.

In the applications.properties we add: quarkus.infinispan-client.hosts and the equivalent env var should be QUARKUS_INFINISPAN_CLIENT_HOSTS.
If we use named configurations, lets say 'london' named connection, i guess the env var should be for quarkus.infinispan-client.london.hosts should be QUARKUS_INFINISPAN_CLIENT_LONDON_HOSTS.
Does this work for you ? @israel-ramos thanks :)

[israel-ramos]

Hi @karesti

Source: https://quarkus.io/guides/infinispan-client-reference

So, I believe the docs are out of date.
Here, we use Quarkus for all our services. As I know, by conventions (Quarkus community), all the configs supported at application.properties could be translated to equivalent ENV var, uppercase, single underscore for dots and double underscore for " (named configs).

We run all our services built to native images (on Kubernetes). So we prefer, as most as possible, to use ENV vars and keep the application.properties empty.

BTW, as a work around I am using as following way for a while.
At application.properties:

quarkus.infinispan-client.tagfull.hosts=${QUARKUS_INFINISPAN_CLIENT__TAGFULL__HOSTS:}
quarkus.infinispan-client.tagfull.username=${QUARKUS_INFINISPAN_CLIENT__TAGFULL__USERNAME:}
quarkus.infinispan-client.tagfull.password=${QUARKUS_INFINISPAN_CLIENT__TAGFULL__PASSWORD:}
quarkus.infinispan-client.use-schema-registration=${QUARKUS_INFINISPAN_CLIENT__TAGFULL__USE_SCHEMA_REGISTRATION:true}

[karesti]

@israel-ramos Sorry for what might be confusing docs, however the docs are correct. The named connections are dependant of your configuration. In this case you named it "tagfull". If you have a single connection, you can avoid naming it for simplicity. https://quarkus.io/guides/infinispan-client-reference#default-and-named-connections

The documentation then lists all the properties. The "named-infinispan-clients" should be replaced by the name you used. If you check any other database that supports naming connections, you will see that all the docs follow that.
However, renaming "named-infinispan-clients" to "infinispan-client-name" might be helpful to avoid confusion. What do you think ? @dmlloyd

[israel-ramos]

@karesti

I have double checked, and none of the following works:

## DOES NOT WORK
QUARKUS_INFINISPAN_CLIENT_TAGFULL_HOSTS=127.0.0.1:11222
QUARKUS_INFINISPAN_CLIENT_TAGFULL_USERNAME=user
QUARKUS_INFINISPAN_CLIENT_TAGFULL_PASSWORD=****
QUARKUS_INFINISPAN_CLIENT_TAGFULL_USE_SCHEMA_REGISTRATION=true

## DOES NOT WORK
QUARKUS_INFINISPAN_CLIENT__TAGFULL__HOSTS=127.0.0.1:11222
QUARKUS_INFINISPAN_CLIENT__TAGFULL__USERNAME=user
QUARKUS_INFINISPAN_CLIENT__TAGFULL__PASSWORD=****
QUARKUS_INFINISPAN_CLIENT__TAGFULL__USE_SCHEMA_REGISTRATION=true

## DOES NOT WORK
QUARKUS_INFINISPAN_CLIENT_HOSTS=127.0.0.1:11222
QUARKUS_INFINISPAN_CLIENT_USERNAME=user
QUARKUS_INFINISPAN_CLIENT_PASSWORD=****
QUARKUS_INFINISPAN_CLIENT_USE_SCHEMA_REGISTRATION=true

PS.: For the last one, not named, I have also removed the annotation @InfinispanClientName("tagfull")
Here, the only way it works is placing the application.properties configs pointing to read from ENV using the ${...}

[karesti]

@israel-ramos when you state "not working" you mean setting the env variables and removing them all from the application.properties, is that alright ? plus, what do you mean by "not working" ? Is there a crash ? An stacktrace on start ? might be helpful to understand the environment where you are testing those, and how (using the operator?) thanks! :)

[israel-ramos]

@karesti
I mean:

jakarta.enterprise.inject.CreationException: Error creating synthetic bean [2V6HFAxu7Ly4lVcHuosbTNsgBtM]: java.lang.IllegalStateException: Unable to produce RemoteCache. RemoteCacheManager is null.

and

Caused by: jakarta.enterprise.inject.CreationException: Null contextual instance was produced by a normal scoped synthetic bean: SYNTHETIC bean [class=org.infinispan.client.hotrod.RemoteCacheManager, id=xtbdHZGBD8crP4969Ppab9uapA4]

Full stacktrace:

jakarta.enterprise.inject.CreationException: Error creating synthetic bean [2V6HFAxu7Ly4lVcHuosbTNsgBtM]: java.lang.IllegalStateException: Unable to produce RemoteCache. RemoteCacheManager is null. Client: health-check
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_Bean.doCreate(Unknown Source)
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_Bean.create(Unknown Source)
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c105(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_ClientProxy.arc$delegate(Unknown Source)
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_ClientProxy.getOrDefault(Unknown Source)
	at digital.ramos.tenantapi.infra.cache.InfinispanCacheCheckController.getValue(InfinispanCacheCheckController.java:94)
	at digital.ramos.tenantapi.infra.cache.InfinispanCacheCheckController_ClientProxy.getValue(Unknown Source)
	at digital.ramos.tenantapi.infra.cache.InfinispanCacheCheckController$quarkusrestinvoker$getValue_2c29e1ba8015728e03586b0580427bf5cbd6ac28.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:599)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: Unable to produce RemoteCache. RemoteCacheManager is null. Client: health-check
	at io.quarkus.infinispan.client.runtime.InfinispanClientProducer.getRemoteCache(InfinispanClientProducer.java:469)
	at io.quarkus.infinispan.client.runtime.InfinispanRecorder$3.apply(InfinispanRecorder.java:54)
	at io.quarkus.infinispan.client.runtime.InfinispanRecorder$3.apply(InfinispanRecorder.java:51)
	at io.quarkus.infinispan.client.runtime.InfinispanRecorder$InfinispanClientSupplier.get(InfinispanRecorder.java:82)
	at io.quarkus.arc.runtime.ArcRecorder$4.apply(ArcRecorder.java:129)
	at io.quarkus.arc.runtime.ArcRecorder$4.apply(ArcRecorder.java:126)
	at org.infinispan.client.hotrod.RemoteCache_2V6HFAxu7Ly4lVcHuosbTNsgBtM_Synthetic_Bean.createSynthetic(Unknown Source)
	... 26 more

[israel-ramos]

@karesti

Just a update.
So far I was using disabled dev services for infinispan QUARKUS_INFINISPAN_CLIENT_DEVSERVICES_ENABLED=false
Because I am using local infinispan cluster (running on local k8s cluster).
So, just for double check, I just tried to re-enable the dev services, and when removing all the env vars, it works.

Do you believe the "dev services" stuff could be impacting the way the configs are being loaded?

PS.: Running here with ./gradlew quarkusDev

[karesti]

@israel-ramos so, you are running infinispan in a local k8s cluster, and your application is dev mode, is that alright ?

[israel-ramos]

@karesti yes, it is. That's the way we setup our local dev env, using a local k8s.

For a few special kind of tests/debug tasks, during development, we depend on other apps running on that cluster to comunicate with the same infinispan within the cluster. The app under development is running on IDE, outside the cluster. That's why it is running using quarkusDev and connecting to infinispan inside the k8s. Also there are other resources, not only other apps, in the same cluster which we depend on like service mesh, gateway classes, http routes, database, etc...

That's the reason.

But anyway, many thanks. We will keep using the workaround at the application.properties doing the config bridge.

[karesti]

@israel-ramos I think in development mode things behave differently. Maybe there is something specific in the Infinispan Extension we are missing for this way of working, but the extension does not do anything in particular and uses quarkus core and config. I'll investigate if this is related to a dev env issue (where the bridge is needed for any extension running this way) or an issue related to the extension.

@geoand maybe you can help clarify if this behaviour is normal in any other extension using a database, for example.

[geoand]

maybe you can help clarify if this behaviour is normal in any other extension using a database, for example

Which behavior are we talking about?

[karesti]

The env variables and dev mode

[israel-ramos]

@geoand This one:

Using:

@Inject
@Remote("health-check")
RemoteCache<String, String> healthCache;

In this way, the application.properties as following, IT IS WORKING, and the remote cache instance is being injected normally:

# IT WORKS
quarkus.infinispan-client.hosts=127.0.0.1:11222
quarkus.infinispan-client.username=user
quarkus.infinispan-client.password=****
quarkus.infinispan-client.use-schema-registration=true

But in the following way, DOES NOT WORK. The remote cache instance IS NOT being injected.

# DOES NOT WORK
quarkus.infinispan-client.devservices.enabled=false
quarkus.infinispan-client.hosts=127.0.0.1:11222
quarkus.infinispan-client.username=user
quarkus.infinispan-client.password=****
quarkus.infinispan-client.use-schema-registration=true

PS.: Just to clear. The Infinispan server running at 127.0.0.1:11222 is NOT the container provided by infinispan client dev mode. This server is intentionally provided as is by myself, with no special config.

[geoand]

I just tried:

QUARKUS_INFINISPAN_CLIENT_HOSTS=localhost:11222 mvn quarkus:dev

and it worked as expected (with the cavear that the logging is misleading which is why I opened #40936).