Error running Consul health check over https with selfsigned CA

[melboyscout]

In my infrustructure used consul like service discovery with self signed certificates.
But rabbitmq do not connect to consul for registering healthcheck.

{"time":"2024-07-04 10:00:27.868589+03:00","level":"error","msg":"Error running Consul health check: "{failed_connect,\n [{to_address,{\"client.volvo.mito\",8501}},\n {inet,\n [inet],\n {tls_alert,\n {unknown_ca,\n \"TLS client: In state wait_cert_cr at ssl_handshake.erl:2127 generated CLIENT ALERT: Fatal - Unknown CA\\n\"}}}]}"","line":560,"pid":"<0.28152.0>","file":"rabbit_peer_discovery_consul.erl","domain":"rabbitmq.peer_discovery","mfa":["rabbit_peer_discovery_consul","send_health_check_pass",0]}
{"time":"2024-07-04 10:00:42.863766+03:00","level":"notice","msg":"TLS client: In state wait_cert_cr at ssl_handshake.erl:2127 generated CLIENT ALERT: Fatal - Unknown CA\n","line":2127,"pid":"<0.28703.0>","file":"ssl_handshake.erl","depth":20,"mfa":["ssl_handshake","path_validation_alert",1]}

rabbit conf:

cluster_name = f1
definitions.import_backend = local_filesystem
definitions.local.path = /etc/rabbitmq/definitions.json
loopback_users.guest = false
hipe_compile = false
listeners.tcp = none
listeners.ssl.default = 5671
stomp.listeners.tcp = none
stomp.listeners.ssl.1 = 61614
stomp.hide_server_info= true
log.console = true
log.console.level = info
log.console.formatter = json
ssl_options.cacertfile = /secrets/ca.pem
ssl_options.certfile = /secrets/cert.pem
ssl_options.keyfile = /secrets/private_key.pem
ssl_options.depth = 2
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.svc_tags.1 = ui
cluster_formation.consul.svc_tags.2 = management
cluster_formation.consul.svc_tags.3 = clustering
cluster_formation.consul.acl_token = ***
cluster_formation.consul.host = client.volvo.mito
cluster_formation.consul.scheme = https
cluster_formation.consul.port = 8501
cluster_formation.consul.svc_addr = {{ env "attr.unique.hostname" }}
cluster_formation.consul.use_longname = true
cluster_formation.consul.svc_ttl = 30
cluster_formation.consul.deregister_after = 90
cluster_partition_handling = autoheal
cluster_formation.consul.svc = rabbitmq-clustering
cluster_formation.consul.svc_port = 15672

a1d81dd65da7:/# openssl x509 –noout –modulus –in /secrets/cert.pem | openssl md5
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
a1d81dd65da7:/#
a1d81dd65da7:/# openssl rsa –noout –modulus –in /secrets/private_key.pem | openssl md5
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
a1d81dd65da7:/#
a1d81dd65da7:/# openssl verify -verbose -CAfile /secrets/ca.pem /secrets/cert.pem
/secrets/cert.pem: OK
a1d81dd65da7:/#
a1d81dd65da7:/# openssl verify -verbose -CAfile /usr/local/share/ca-certificates/volvo.crt /secrets/ca.pem
/secrets/ca.pem: OK
a1d81dd65da7:/#
a1d81dd65da7:/# openssl verify -verbose -CAfile /usr/local/share/ca-certificates/mito.crt /usr/local/share/ca-certificates/volvo.crt
/usr/local/share/ca-certificates/volvo.crt: OK

a1d81dd65da7:/# curl -v client.volvo.mito:8501

• Host client.volvo.mito:8501 was resolved.
  ...
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
  ...
• Server certificate:
• subject: CN=client.volvo.mito
  ...
• SSL certificate verify ok.

[LaurentGoderre]

Is the consul certificate mounted in the container?

[lukebakken]

@melboyscout this repository is NOT the correct place to ask general RabbitMQ questions. Please ask in one of these forums:

• https://groups.google.com/g/rabbitmq-users
• https://github.com/rabbitmq/rabbitmq-server/discussions

In addition, please review Team RabbitMQ's community support policy. At the very least, you must provide a reproducible example that we can run to see this issue.

I'm pretty sure your solution will be to set the correct cluster_formation.consul.ssl_options values, which don't appear to be documented

• https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-consul
• https://github.com/rabbitmq/rabbitmq-server/pull/5155/files

Note that the ssl_options values available are the same as for etcd

[lukebakken]

For reference: rabbitmq/rabbitmq-website#567