Connectors fails to authenticate to operate: Target host not specified

[jessesimpson36]

Describe the bug

In #928 , I described an issue where I could not use the docker compose deployment method on the latest 8.4 release of connectors. This issue is quite similar, but with a different stacktrace.

I think the patch @jonathanlukas supplied in
#928 (comment)

was helpful at getting around the first stacktrace, but this I feel is different.

Things that I've tried with no luck:

1. Setting CAMUNDA_OPERATE_CSRFPREVENTIONENABLED=false and CAMUNDA_TASKLIST_CSRFPREVENTIONENABLED=false environment variables in operate and tasklist respectively
2. Setting operate to be 8.4.11 along with connectors 8.4.10
3. Building a set of jar files of spring-zeebe with the patch @jonathanlukas suggested, and building a connectors jar using that built dependency of spring-zeebe, along with building connectors docker image and using that with the docker compose file.

However, the error remains: Caused by: org.apache.hc.core5.http.ProtocolException: Target host is not specified

I suspect that it's not even trying to send an http request, because when I view the connectors pod in wireshark, I see no http requests.

To Reproduce

Expected behavior

Log/Stacktrace

Full Stacktrace

2024-09-05T02:08:41.397Z ERROR 1 --- [   scheduling-1] i.c.common.auth.SimpleAuthentication     : Authenticating for OPERATE failed due to org.apache.hc.client5.http.ClientProtocolException: Target host is not specified
2024-09-05T02:08:41.397Z ERROR 1 --- [   scheduling-1] i.c.c.r.i.i.ProcessDefinitionImporter    : Failed to import process definitions

java.lang.RuntimeException: Unable to authenticate
	at io.camunda.common.auth.SimpleAuthentication.retrieveToken(SimpleAuthentication.java:54)
	at io.camunda.common.auth.SimpleAuthentication.getTokenHeader(SimpleAuthentication.java:84)
	at io.camunda.common.http.DefaultHttpClient.retrieveToken(DefaultHttpClient.java:207)
	at io.camunda.common.http.DefaultHttpClient.post(DefaultHttpClient.java:148)
	at io.camunda.operate.CamundaOperateClient.searchProcessDefinitionResults(CamundaOperateClient.java:46)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:72)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:55)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124)
	at io.micrometer.observation.Observation.observe(Observation.java:499)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hc.client5.http.ClientProtocolException: Target host is not specified
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:173)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:162)
	at io.camunda.common.auth.SimpleAuthentication.retrieveToken(SimpleAuthentication.java:39)
	... 19 common frames omitted
Caused by: org.apache.hc.core5.http.ProtocolException: Target host is not specified
	at org.apache.hc.client5.http.impl.routing.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:66)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.determineRoute(InternalHttpClient.java:120)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:158)
	... 23 common frames omitted

Environment:

• OS:
• Deployment:
• Spring-Zeebe Version:
• SpringBoot Version:
• Configuration:

[jessesimpson36]

I think I may have figured it out. it appears #662 changed the configuration properties, and adding these env vars helped:

+      - CAMUNDA_CLIENT_MODE=simple
+      - CAMUNDA_CLIENT_AUTH_USERNAME=demo
+      - CAMUNDA_CLIENT_AUTH_PASSWORD=demo
+      - CAMUNDA_CLIENT_OPERATE_BASE_URL=http://operate:8080
+      - CAMUNDA_CLIENT_TASKLIST_BASE_URL=http://tasklist:8080

[jessesimpson36]

I'll try to get back to this topic tomorrow. I want to leave the github issues open for now until I get this patched up downstream (compose and maybe helm).

[jonathanlukas]

With the changes from #662 there is only 1 property to add:

+ CAMUNDA_OPERATE_CLIENT_BASEURL=http://operate:8080

Other than that, the settings for the oidc connection could be off a bit more. For 8.5, the configuration has been migrated already, maybe we can copy from there.

[jessesimpson36]

@jonathanlukas that parameter has been around since 2022. After #662 , that parameter no longer works in 8.4.9 of connectors in docker-compose-core.yaml :

spring-zeebe/spring-boot-starter-camunda/src/main/java/io/camunda/zeebe/spring/client/properties/OperateClientConfigurationProperties.java

Lines 87 to 92 in ef37537

	@DeprecatedConfigurationProperty(replacement = "camunda.client.operate.base-url")
	@Deprecated
	public String getUrl() {
	return url;
	}

The new parameter is CAMUNDA_CLIENT_OPERATE_BASE_URL. Was mentioning CAMUNDA_OPERATE_CLIENT_BASEURL a mistake? or is that parameter expected to work, but doesn't (indicating a code-bug / and that I should not patch docker-compose-core.yaml).

[jonathanlukas]

I got it working locally :)

Reason: CAMUNDA_OPERATE_CLIENT_BASEURL refers to:

spring-zeebe/spring-boot-starter-camunda/src/main/java/io/camunda/zeebe/spring/client/properties/OperateClientConfigurationProperties.java

Line 34 in ef37537

private String baseUrl;

which is then taken as base url when configuring the authentication:

spring-zeebe/spring-boot-starter-camunda/src/main/java/io/camunda/zeebe/spring/client/configuration/CommonClientConfiguration.java

Line 79 in ef37537

operateClientConfigurationProperties.getBaseUrl(),

[jessesimpson36]

When you say that you got it working locally, this kinda just sounds like you weren't able to reproduce my bug. Did you make a code change to make it work? or are you claiming that it always worked despite my bug report?

[jonathanlukas]

Using the latest 8.4 connectors-bundle works when adding this one env property :) no bug, just a messy config

[jessesimpson36]

Ok, but adding that env property CAMUNDA_OPERATE_CLIENT_BASEURL does not work for me when I deploy from the docker-compose-core.yaml file on newer versions of connectors (8.4.9+).

Was your test setup an intellij / java-only install method ?

[jessesimpson36]

After discussing on a call with @jonathanlukas , we learned that I was improperly testing the recommended workaround by not having both

CAMUNDA_OPERATE_CLIENT_BASEURL

and

CAMUNDA_OPERATE_CLIENT_URL

set at the same time. (in addition to the CSRF variable). So we have 2 configuration syntax's , the old way (using CAMUNDA_OPERATE_CLIENT_BASEURL) and the new way using CAMUNDA_CLIENT_OPERATE_BASE_URL . and so far we think the best thing to do with affected areas of compose / helm is to get things working using the old method, since we expect 8.6 will change other aspects of how components connect to eachother.

However, at this point, the only affected environment appears to be connectors 8.4.9 through connectors 8.4.11 with docker-compose on stable/8.4. newer compose files don't have this issue for some unknown reason. If we find other environments that are broken for this reason, we can come back to this issue and re-assess whether we should migrate to the new configuration syntax or if an additional code-change should be added.