Consul goroutines overwhelmed

[jacobomarquis]

We enabled Consul Service Discovery and Spring starts opening connections to the Consul Server never closing and exhausting available connections. We dont see anything special on the logs, but on the systems side (using netstat) we see a 500+ "ESTABLISED" connections from the Spring microservice.

Any ideas?

Thanks in advance

[spencergibb]

No, unfortunately. Any more information you can provide? Versions? Configuration?

[jacobomarquis]

We observed this happening in Spring 1 and Spring 2.
This is our config:

spring:
  cloud:
    discovery:
      enabled: true
    consul:
      enabled: true
      host: ${consul_host:https://consul}
      port: ${consul_port:8500}
      discovery:
        enabled: true
        register: false
        instance-id: ${spring.application.name}:${random.value}
        catalog-services-watch-delay: 60000
        catalog-services-watch-timeout: 10
        scheme: https
      tls:
        key-store-instance-type: ${kstore_instance_type}
        key-store-path: ${kstore_path}
        key-store-password: ${kstore_password}
        certificate-path: ${cert_path}
        certificate-password: ${cert_password}

All Spring services are running inside an Alpine:3.8 + Java 8 docker container image.

Consul server is running in the Docker Host and accessed through its private IP (not docker network IP)

[jacobomarquis]

We register the service on the orchestrator (Nomad) hence the register: false

[spencergibb]

Spring 1 and Spring 2 are not any versions I recognize. Can you show your dependencies? Nothing looks out of the ordinary.

[franHousers]

We have many services with different spring versions like 1.5.19 and 2.1.3

[franHousers]

Ok, we finally found the bug.

The problem was using the scheme 'https' with certificates, seems that the apache library "org.apache.http" doesn't does the relese connection properly in the package

package org.apache.http.impl.execchain;
Class ConnectionHolder
releaseConnection

It keeps alive indefinitely.

We see the next logs doing the catalogServicesWatch.
o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely

`[TaskScheduler-1] o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely
[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection [id: 3][route: {s}->https://consul.cloud.XXXX.XX:8500][state: CN=agent.global.consul, OU=Cloud, O=XXXXX, L=XXXXX, ST=XXXXX, C=XXXXX] can be kept alive indefinitely
[TaskScheduler-1] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-3: set socket timeout to 0
[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 3][route: {s}->https://consul.cloud.XXXX.XX:8500][state: CN=agent.global.consul, OU=Cloud, O=XXXXX, L=XXXXX, ST=XXXXX, C=XXXXX][total kept alive: 4; route allocated: 4 of 500; total allocated: 4 of 1000]
[TaskScheduler-1] o.s.c.c.discovery.ConsulDiscoveryClient : Received services update from consul: {consul=[], nomad=[http, serf, rpc]}, index: 55375

[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection request: [route: {s}->https://consul.cloud.XXXXX.XX:8500][total kept alive: 4; route allocated: 4 of 500; total allocated: 4 of 1000]
[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection leased: [id: 4][route: {s}->https://consul.cloud.XXXXX.XX:8500][total kept alive: 4; route allocated: 5 of 500; total allocated: 5 of 1000]
[TaskScheduler-1] o.a.http.impl.execchain.MainClientExec : Opening connection {s}->https://consul.cloud.XXXXX.XX:8500

[TaskScheduler-1] o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely
[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection [id: 4][route: {s}->https://consul.cloud.XXXXX.XX:8500][state: CN=agent.global.consul, OU=Cloud, O=XXXXX, L=XXXXX, ST=XXXXX, C=ES] can be kept alive indefinitely
[TaskScheduler-1] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-4: set socket timeout to 0
[TaskScheduler-1] h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 4][route: {s}->https://consul.cloud.XXXXX.XX:8500][state: CN=agent.global.consul, OU=Cloud, O=XXXXX, L=XXXXX, ST=XXXXX, C=ES][total kept alive: 5; route allocated: 5 of 500; total allocated: 5 of 1000]
[TaskScheduler-1] o.s.c.c.discovery.ConsulDiscoveryClient : Received services update from consul: {consul=[], nomad=[rpc, http, serf]}, index: 55375`

[spencergibb]

Can I close this then?

[jacobomarquis]

There's still an issue when Springs is configured to use SSL connection to Consul. We worked around the problem using plain HTTP

[spencergibb]

We do very little to configure the ConsulClient.

spring-cloud-consul/spring-cloud-consul-core/src/main/java/org/springframework/cloud/consul/ConsulAutoConfiguration.java

Lines 55 to 69 in f10cb55

	public ConsulClient consulClient(ConsulProperties consulProperties) {
	final int agentPort = consulProperties.getPort();
	final String agentHost = !StringUtils.isEmpty(consulProperties.getScheme())
	? consulProperties.getScheme() + "://" + consulProperties.getHost()
	: consulProperties.getHost();
	if (consulProperties.getTls() != null) {
	ConsulProperties.TLSConfig tls = consulProperties.getTls();
	TLSConfig tlsConfig = new TLSConfig(tls.getKeyStoreInstanceType(),
	tls.getCertificatePath(), tls.getCertificatePassword(),
	tls.getKeyStorePath(), tls.getKeyStorePassword());
	return new ConsulClient(agentHost, agentPort, tlsConfig);
	}
	return new ConsulClient(agentHost, agentPort);
	}

My guess is that you need to open an issue here https://github.com/Ecwid/consul-api

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[rola0411]

@jacobomarquis

I want to ask a question. I have ca.pem and ca-key.pem files

But I don't know how to get the things in these variables.

for example

key-store-instance-type:  ??? in the springboot ，Which one to choose ： JKS,JCEKS, PKCS12, PKCS11, DKS;
key-store-path: ??? what is this ? How can i get it
key-store-password: ??? what is this ? How can i get it
certificate-path: consul-client-ca.pem
certificate-password: ??? what is this ? How can i get it
```y

Could you tell me. Thank you very much