Setting the TLS SNI hostname

[issacg]

Hi,

I have a vault instance (server + injector) setup by the Helm chart. The server is accessible from outside the cluster, and so the TLS setup on the vault server uses a certificate for vault.domain.com signed by LetsEncrypt - so far so good!

I'm trying to set up the agent injector to work with a an internal pod. Internally, the injector is trying to connect to vault.my-namespace.svc and the agent is filled with messages like: 2020/06/16 16:28:54.677852 [WARN] (view) vault.read(auth/token/lookup-self): vault.read(auth/token/lookup-self): Get https://vault.namespace.svc/v1/auth/token/lookup-self: x509: certificate is valid for vault.domain.com, not vault.namespace.svc (retry attempt 5 after "4s")

I found the annotation tls-server-name which at first glance seems to be intended to fix exactly this case, but setting the annotation vault.hashicorp.com/tls-server-name: vault.domain.com still results in the same log entries.

Am I on the right track? Is there a right way to do this that won't require setting tls-skip-verify which I never like to have to do?

Thanks!

[AaronForce1]

Hey @issacg - I haven't gotten as far as you... I'm still trying to configure the helm chart so that outside cluster access is possible and TLS is configured by letsencrypt-prod. How did you configure the requisite svc/ingress to do so? Any assistance is appreciated.

[issacg]

This is a bit OT, but here's the short version.

I used cert-manager to issue the keypair and then used server.config in the Helm chart as follows:

    config: |
      listener "tcp" {
        tls_cert_file = "/vault/tls/the-certmanager-secret/tls.crt"
        tls_key_file = "/vault/tls/the-certmanager-secret/tls.key"
        /* other stuff*/
    }

and server.extraVolumes as follows:

 extraVolumes:
    - type: secret
      name: the-certmanager-secret
      path: /vault/tls

For the service, you can either expose it directly or, if you want, use an ingress proxy or a LoadBalancer. If you want end-to-end encryption (which is the recommended best-practice), use a level 4 (network) load balancer or ingress proxy. Be aware that using a level 4 proxy or load balancer may cause you to lose the chain of trust for X-Forwarded-For headers, if you count on them.

LMK if it's still not clear; it's a great idea for a blog post, if I can find the time to make one this week...

[AaronForce1]

Yea - I was thinking of going down the same path, but did you configure cert-manager to request certificates via dns-01 and then you applied the keys/certs into k8s? Or did you do something supplemental before applying the helm configuration in order to get letsencrypt-prod verified as vault pods were deployed?

[AaronForce1]

@issacg So I was able to manage using a custom-defined k8s ingress (using cert-manager/letsencrypt-prod) which then pointed to a non-TLS vault (UI service at port 80). Is this the best approach or is it more recommended to ensure all of vault is running as TLS, versus just an nginx-ingress?

[issacg]

@AaronForce1 as promised, https://www.linkedin.com/pulse/vaults-tls-es-k8s-es-ingress-es-oh-my-issac-goldstand/

Let's move the conversation there if you have any follow-ups, so that we can keep this ticket on-topic!

[pbar1]

Hmm interesting - I stumbled across this thread just by chance, so I decided to test it (we use the cert auth method with the Agent+Injector). I tried setting the annotation:

vault.hashicorp.com/tls-server-name: vault-active.vault.svc.cluster.local

...which (correctly) results in the following agent config (abbreviated for conciseness):

{
  "auto_auth": {
    "method": {
      "type": "cert",
      "mount_path": "auth/cert",
      "config": {
        "name": "test-cert"
      }
    },
    "sink": <snip>,
  "exit_after_auth": false,
  "pid_file": "/home/vault/.pid",
  "vault": {
    "address": "https://vault.our-namespace.svc.clustername.com",  <-- Vault server's ingress addr, which is a valid SAN
    "ca_cert": "/certs/cacerts.pem",
    "client_cert": "/certs/client.pem",
    "client_key": "/certs/client-key.pem",
    "tls_server_name": "vault-active.vault.svc.cluster.local"  <-- also a valid SAN on our cert, so also correct
  },
  "template": <snip>
}

results in a debug log line 2021/02/13 01:21:15.832010 [DEBUG] (runner) final config: like so:

{
    # ...snip
    "Vault": {
        "Address": "https://vault.our-namespace.svc.clustername.com", 
        "Enabled": true,
        "Namespace": "",
        "RenewToken": false,
        "Retry": <snip>,
        "SSL": {
            "CaCert": "/certs/cacerts.pem",
            "CaPath": "",
            "Cert": "/certs/client.pem",
            "Enabled": true,
            "Key": "/certs/client-key.pem",
            "ServerName": "",  <-- notice this is blank
            "Verify": true
        },
    # snip...
}

So, it's possible that ServerName doesn't get set? I have not read the logic yet however, so this could just be expected.

Edit: looks like that debug line is generated by Consul Template - my current theory is that it's dropped when calling CT.

[pbar1]

Yep, looks like Consul Template's ServerName field isn't being set here. This is an issue for hashicorp/vault tho, so this one can probably be closed.