Vault fails to supply x509 certificate when connecting to Consul when mutual TLS is enabled #3248
Comments
|
You're claiming that Vault is not sending certificates to Consul. How have you verified this? The error message says it's a bad certificate, not that there is no certificate. |
|
Hi @jefferai , Thanks for the reply! Sorry for the delay in getting back to you. I approached the error message by first verifying the x509 certificates being used by Vault and Consul would accept a connection from each other (using the I used the Consul certificate for a "server" process requiring mutual TLS: openssl s_server -CAfile agent.ca.pem -cert agent.cert.pem -key agent.key.pem -Verify 1 -wwwI used the Vault certificate as the "client". openssl s_client -CAfile server.ca.pem -cert server.cert.pem -key server.key.pem -verify 1 -connect localhost:4433The openssl server/client interaction demonstrated that the certificates were capable of mutual TLS. The server transcript is as follows. The client transcript is as follows. Seeing the certificates work with one another, I then updated my docker-compose file to run an This is the substituted command. This is the information recorded by As such, I reasoned that Vault was not responding to the client certificate request. Not knowing how to proceed, I updated the local Consul agent configuration (which is dedicated to the Vault process) to The repository to which I linked was a small testbed that I hoped would be of use in quickly reproducing the problem I'm encountering. If there is anything else I can do or provide to aid in the diagnosis of the problem I've encountered please don't hesitate to ask. Thanks!
|
|
Hi Simon , Looking at your testing methodology I still think that the problem isn't that Vault's not supplying a certificate, it's that the remote end is failing to verify it. (Note that the error message says "bad certificate", not "no certificate".) When you are using curl you are explicitly specifying a CA certificate for the client. My guess is that if you left that off you'd see the same behavior. The solution is likely to be concatenating your CA certificate into your client certificate file. It should look like this: Let me know how it goes! |
|
Hi @jefferai , The client certificates being used by Vault and Consul are the concatenated In my Consul entrypoint. cat <<-EOF > "${pki_request}"
{
"common_name": "$(hostname -f)",
"alt_names": "localhost",
"ttl": "8760h",
"ip_sans": "$(hostname -i),127.0.0.1",
"format": "pem"
}
EOF
pki=$(curl --insecure --fail --request POST --header "X-Vault-Token: ${vault_token}" --data @"${pki_request}" "${VAULT_ADDR}/v1/pki-ca/issue/tls-cert")
echo "${pki}" | jq -r .data.certificate > "${CONSUL_CERT_FILE}"
echo "${pki}" | jq -r .data.issuing_ca >> "${CONSUL_CERT_FILE}"
echo "${pki}" | jq -r .data.private_key > "${CONSUL_KEY_FILE}"
echo "${pki}" | jq -r .data.issuing_ca > "${CONSUL_CA_FILE}"In my Vault entrypoint. pki=$(vault write -tls-skip-verify -address="${CA_VAULT_ADDR}" -format=json "pki-ca/issue/tls-cert" "common_name=$(hostname -f)" "alt_names=localhost" "ttl=8760h" "ip_sans=$(hostname -i),127.0.0.1" "format=pem")
echo "${pki}" | jq -r .data.certificate > "${VAULT_CERT_FILE}"
echo "${pki}" | jq -r .data.issuing_ca >> "${VAULT_CERT_FILE}"
echo "${pki}" | jq -r .data.private_key > "${VAULT_KEY_FILE}"
echo "${pki}" | jq -r .data.issuing_ca > "${VAULT_CA_FILE}"I wrote the following short program to try and isolate/reproduce package main
import "github.com/hashicorp/consul/api"
import "fmt"
func main() {
client, err := api.NewClient(api.DefaultConfig())
if err != nil {
panic(err)
}
checks, _, err := client.Health().State("any", &api.QueryOptions{})
if err != nil {
panic(err)
}
for _, element := range checks {
fmt.Printf("{'Node': '%s', 'Status': '%s', 'Name': '%s'}\n", element.Node, element.Status, element.Name)
}
}I ran the program as follows on the Vault host. export CONSUL_HTTP_ADDR="127.0.0.1:8500"
export CONSUL_HTTP_SSL="true"
export CONSUL_HTTP_SSL_VERIFY="true"
export CONSUL_CACERT="/opt/vault/config/keys/server.ca.pem"
export CONSUL_CLIENT_CERT="/opt/vault/config/keys/server.cert.pem"
export CONSUL_CLIENT_KEY="/opt/vault/config/keys/server.key.pem"
export CONSUL_TLS_SERVER_NAME="consul-client.dc.consul"
./mainIt printed the following text to the terminal. Since I received a response (and not a My next experiment was to rely only on the environment variables for the {
"storage": {
"consul": {
"cluster_addr": "https://172.19.0.12:8201",
"redirect_addr": "https://172.19.0.12:8200"
}
}
}The Vault startup logs now show another error ( This was interesting as I was expecting it to work (like the test program). I then updated the configuration to specify the CA file (as the environment variable {
"storage": {
"consul": {
"cluster_addr": "https://172.19.0.12:8201",
"redirect_addr": "https://172.19.0.12:8200",
"tls_ca_file": "/opt/vault/config/keys/server.ca.pem"
}
}
}The error reverted to My next experiment was to add the CA pem to the system bundle (and removing, once cp "${VAULT_CA_FILE}" /etc/pki/ca-trust/source/anchors/ && update-ca-trust extractand cp "${CONSUL_CA_FILE}" /etc/pki/ca-trust/source/anchors/ && update-ca-trust extractfor the Vault and Consul services (respectively). This also produced the error I then reviewed the difference between the test code and Vault
where the test code uses the latter and Vault the former. I cannot detect any As such, I updated the test program to configure its connection with Consul in package main
import (
"fmt"
"net/http"
"golang.org/x/net/http2"
"github.com/hashicorp/consul/api"
"crypto/tls"
)
func main() {
config := api.DefaultConfig()
config.Transport.MaxIdleConnsPerHost = 64
tlsClientConfig, err := api.SetupTLSConfig(&config.TLSConfig)
if err != nil {
panic(err)
}
tlsClientConfig.MinVersion = tls.VersionTLS12
config.Transport.TLSClientConfig = tlsClientConfig
if err := http2.ConfigureTransport(config.Transport); err != nil {
panic(err)
}
config.HttpClient = &http.Client{Transport: config.Transport}
fmt.Printf("%#v\n", config)
client, err := api.NewClient(config)
if err != nil {
panic(err)
}
checks, _, err := client.Health().State("any", &api.QueryOptions{})
if err != nil {
panic(err)
}
for _, element := range checks {
fmt.Printf("{'Node': '%s', 'Status': '%s', 'Name': '%s'}\n", element.Node, element.Status, element.Name)
}
_, err = client.KV().Put(&api.KVPair{Key:"hello", Value: []byte("world")}, &api.WriteOptions{})
if err != nil {
panic(err)
}
pair, _, err := client.KV().Get("hello", &api.QueryOptions{})
if err != nil {
panic(err)
}
fmt.Printf("%s=%s\n", pair.Key, string(pair.Value))
}The program produces the following output. I have, for the time being, ran out of ideas that might lead to a fix for the Do you have any recommendations? Thanks!
|
|
Hi Jeff, I added logging to a development build of Vault and found that Best, Simon |
|
Did you ever try using tls_ca_file to specify the CA cert in conjunction with the client certificate? |
|
I'm a bit confused by what's going on here...you opened a PR but now closed it...is it working for you? |
|
Hi Jeff, I found my mistake -- really quite embarrassing. After having added logging to I've submitted my logging changes as a PR on the off chance I'm not the only one who will make this mistake. Thanks for your help!
|
|
@simon-wenmouth hah, okay -- I will look at the PR. I acutally looked earlier at the way that Consul was setting up TLS in their function and ours and was like...I don't really see why there should be a difference :-) |
|
Closing, will look at the PR. |
Summary
verify_incoming: trueand
verify_outgoing: truePUT /v1/agent/service/register)during the TLS handshake Consul sends a client certificate request.
Expected Behavior:
Vault should respond with its certificate during the mutual TLS handshake with Consul.
Actual Behavior:
Vault did not reply with its certificate when responding to the client certificate request.
Steps to Reproduce:
See: https://github.com/simon-wenmouth/vault-consul-tls/
Environment:
Vault Config File:
See: https://github.com/simon-wenmouth/vault-consul-tls/tree/master/etc/vault
{ "cluster_name": "intermediate-ca", "cache_size": "32000", "disable_cache": false, "disable_mlock": true, "default_lease_ttl": "768h", "max_lease_ttl": "768h" }{ "listener": { "tcp": { "address": "0.0.0.0:8200", "cluster_address": "172.19.0.12:8201", "tls_disable": "false", "tls_cert_file": "/opt/vault/config/keys/server.cert.pem", "tls_key_file": "/opt/vault/config/keys/server.key.pem" } } }{ "storage": { "consul": { "address": "consul-client.dc.consul:8500", "check_timeout": "5s", "consistency_mode": "default", "disable_registration": "false", "max_parallel": "128", "path": "vault/", "scheme": "https", "service": "vault", "service_tags": "", "token": "", "cluster_addr": "https://172.19.0.12:8201", "redirect_addr": "https://172.19.0.12:8200", "tls_ca_file": "/opt/vault/config/keys/server.ca.pem", "tls_cert_file": "/opt/vault/config/keys/server.cert.pem", "tls_key_file ": "/opt/vault/config/keys/server.key.pem" } } }Startup Log Output:
The text was updated successfully, but these errors were encountered: