Skip to content

Hashicorp Vault Keystore

Jump to bottom
Andreas Auernhammer edited this page Apr 8, 2020 · 33 revisions

How to use Hashicorp Vault as key store

This guide shows how to setup a KES server that uses Vault's K/V engine as a persistent and secure key store:

  1. Vault Server Setup
  2. KES Server Setup
  3. Recommendations
                              ┌───────┐      
                              |  KMS  ├─────┐  
                              └───┬───┘     |  
                                  |         |  
                         ╔════════╪═════════╪══════════════════════════╗
┌────────────┐           ║  ┌─────┴──────┐  |           ┌───────────┐  ║
│ KES Client ├───────────╫──┤ KES Server ├──┴───────────┤   Vault   │  ║
└────────────┘           ║  └────────────┘              └───────────┘  ║
                         ╚═════════════════════════════════════════════╝

Vault Server Setup

  1. Generate a TLS certificate for Vault

    KES and Vault will exchange sensitive information. In particular, KES will send and receive the secret keys from Vault's HTTP API. Therefore, it is necessary to secure the communication between KES and Vault via TLS. Here, we use self-signed certificates for simplicity. For a production setup we highly recommend certificates signed by CA (e.g. your internal CA or a public CA like Let's Encrypt).

    • To generate a new private key for Vault's certificate run the following openssl command: 
      openssl ecparam -genkey -name prime256v1 | openssl ec -out vault-tls.key
    • Then generate a new TLS certificate for the private/public key pair via: 
      openssl req -new -x509 -days 30 -key server.key -out server.cert \
  -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"

      Here, we assume that we will run Vault locally as https://127.0.0.1:8200. If you want to run Vault on a different host you may adjust the CN and SAN accordingly.

  2. Download and start a Vault Server

    If you haven't already, download the Vault binary. To avoid running Vault as root we must grant the binary the ipc_lock capability such that it can use the mlock syscall:

    sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

    Then you can use the following Vault configuration to start a Vault server on https://127.0.0.1:8200:

    cat > vault-config.json <<EOF
{
   "api_addr": "https://127.0.0.1:8200",
   "backend": {
     "file": {
       "path": "vault/file"
     }
   },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "vault-tls.crt",
      "tls_key_file": "vault-tls.key",
      "tls_min_version": "tls12"
    }
  }
}
EOF

    Note that we run Vault with a file backend. For high-availability in a real production environment you may want to use etcd, consul or Vault with integrated storage instead.

    Start the Vault server via:

    vault server -config vault-config.json

  3. Initialize and unseal Vault

    In a separate terminal window set the VAULT_ADDR env. variable to your Vault server:

    export VAULT_ADDR='http://127.0.0.1:8200'

    Further, you may want to run export VAULT_SKIP_VERIFY=true if Vault uses a self-signed TLS certificate. When Vault serves a TLS certificate that has been issued by a CA that is trusted by your machine - e.g. Let's Encrypt - then you don't need to run this command.

    Then initialize Vault via:

    vault operator init

    Vault will print n (5 by default) unseal key shares of which at least m (3 by default) are required to regenerate the actual unseal key to unseal Vault. Therefore, make sure to remember them. In particular, keep those unseal key shares at a secure and durable location.

    You should see some output similar to:

       Unseal Key 1: eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
   Unseal Key 2: 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
   Unseal Key 3: cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
   Unseal Key 4: /fTPpec5fWpGqWHK+uhnnTNMQyAbl5alUi4iq2yNgyqj
   Unseal Key 5: UPdDVPto+H6ko+20NKmagK40MOskqOBw4y/S51WpgVy/

   Initial Root Token: s.zaU4Gbcu0Wh46uj2V3VuUde0

   Vault is initialized with 5 key shares and a key threshold of 3. Please securely
   distribute the key shares printed above. When the Vault is re-sealed,
   restarted, or stopped, you must supply at least 3 of these keys to unseal it
   before it can start servicing requests.

    Now, set the env. variable VAULT_TOKEN to the root token printed by the command before:

    export VAULT_TOKEN=s.zaU4Gbcu0Wh46uj2V3VuUde0

    Then use any of the previously generated key shares to unseal Vault.

    vault operator unseal eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
vault operator unseal 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
vault operator unseal cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF

    Once you have submitted enough valid key shares Vault will become unsealed and able to process requests.

  4. Enable Vault's K/V backend

    KES will store the secret keys at the Vault K/V backend.

    vault secrets enable kv

  5. Enable AppRole authentication

    KES supports Vault's AppRole authentication mechanism.

    vault auth enable approle

  6. Create an access policy for the K/V engine

    KES will use an AppRole ID that has the following policy permissions.

    cat > kes-policy.hcl <<EOF
path "kv/*" {
     capabilities = [ "create", "read", "delete" ]
}
EOF

    If you don't want to create or delete keys using the KES server but want to manage them directly using e.g. the Vault CLI (vault kv --help) you may use capabilities = [ "read" ] instead.

    Then we upload the policy to Vault:

    vault policy write key-policy ./kes-policy.hcl

  7. Create an new AppRole ID and bind it to a policy

    Now, we need to create a new AppRole ID with the required permissions. The application (i.e. KES server) will authenticate to Vault via the AppRole role ID and secret ID and is only allowed to perform operations granted by the kes-policy.hcl policy. So, we first create a new role for our KES server:

    vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m

    Then we bind the kes-policy policy to the role:

    vault write auth/approle/role/kes-role policies=key-policy

    Finally, we request an AppRole role ID and secret ID from Vault.
    First, the role ID:

    vault read auth/approle/role/kes-role/role-id

    Then, the secret ID:

    vault write -f auth/approle/role/kes-role/secret-id

    We are only interested in the secret_id - not in the secret_id_accessor.

KES Server Setup

Similar to Vault, KES can use the mlock syscall on linux systems to prevent the OS from writing in-memory data to disk (swapping). Therefore, you should give the KES executable the ability to use the mlock syscall without running the process as root. To do so run:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

First we need to generate a TLS private key and certificate for our KES server. A KES server can only be run with TLS - since secure-by-default. Here we use self-signed certificates for simplicity. For a production setup we highly recommend to use a certificate signed by CA (e.g. your internal CA or a public CA like Let's Encrypt)

  1. First, create the TLS private key:

    openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key

  2. Then, create the corresponding TLS X.509 certificate:

    openssl req -new -x509 -days 30 -key server.key -out server.cert \
  -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"

  3. Now you have a server.key and server.cert file. Next, we create the root identity:

    kes tool identity new --key="root.key" --cert="root.cert" root

    Note that we create a private key (root.key) and a certificate (root.cert) for TLS client authentication. Again, the certificate is not signed by a CA that is trusted by the KES server. That is not a security issue per se since only clients with public keys/certificates that are known to the server can perform operations based on policies. However, we recommend to use client certificates that were issued by a trusted CA. Then the kes server does not even accept connections from untrusted clients.

    You can compute the root identity via:

    kes tool identity of root.cert

  4. Since we don't want to give our applications root capabilities we also create an app identity:

    kes tool identity new --key="./app.key" --cert="app.cert" app

    You can compute the app identity via:

    kes tool identity of app.cert

  5. Now we have defined all entities in our demo setup. Let's wire everything together by creating the config file server-config.toml:

    address = "127.0.0.1:7373"
root = "<kes-tool-identity of root.cert>"

[tls]
key  = "server.key"
cert = "server.cert"

[policy.prod-app] 
paths      = [ "/v1/key/create/app-key", "/v1/key/generate/app-key" , "/v1/key/decrypt/app-key" ]
identities = [ "<kes-tool-identity-of-app.cert>" ]

[keystore.vault]
address = "https://127.0.0.1:8200"

[keystore.vault.approle]
id     = "" # Your AppRole ID
secret = "" # Your AppRole Secret ID
retry  = "15s"

[keystore.vault.status]
ping = "10s"

[keystore.vault.tls]
ca = "./vault-tls.crt" # Since we use self-signed certificates

    Please use your own root and app identity and AppRole credentials.

  6. Finally we can start a KES server in a new window/tab:

    kes server --mlock --config=server-config.toml --mtls-auth=ignore

    --mtls-auth=ignore is required since our root.cert and app.cert certificates are self-signed.
    The --mlock option is currently only available on Linux.

  7. In the previous window/tab we now can connect to the server by:

    export KES_CLIENT_TLS_CERT_FILE=app.cert
export KES_CLIENT_TLS_KEY_FILE=app.key
kes key create app-key -k

    -k is required because we use self-signed certificates

    Now, you should see a secret key inside the ./keys directory.

  8. Finally, we can derive and decrypt data keys from the previously created app-key:

    kes key derive app-key -k
{
  plaintext : ...
  ciphertext: ...
}
    kes key decrypt app-key -k <base64-ciphertext>

Recommendations

Here we show some additional configuration steps that can solve specific problems.

1. Control who can create secret keys.

In general, the kes server as well as Vault can create and delete secret keys. The kes server exposes the /v1/key/create and v1/key/decrypt REST API. Vault's K/V store exposes an API for creating and deleting secrets.

Now, you can control which one can create (or delete) secret keys using the kes-policy.hcl. If you want to e.g. grant the kes server the permission to create and delete keys from Vault's K/V store then use the following policy:

path "kv/*" {
   capabilities = [ "create", "read", "delete" ]
}

However, you can also grant only the permission to create but not to delete keys using:

path "kv/*" {
   capabilities = [ "create", "read" ]
}

You should not grant the kes server the permission to "update" secret keys. It's not a security issue if you do but it's a) not necessary and b) it can lead to data corruption in distributed systems.

This policy only specifies whether the kes server, in general, can create or delete K/V entries. However, without a policy that explicitly grants access to the /v1/key/create or /v1/key/delete API, only the root identity can perform those operations. Therefore, you can define a kes server policy that allows creating and deleting secret keys that start with my-key-:

 cat > create-and-delete-policy.toml <<EOF 
 paths = ["/v1/key/create/my-key-*", "/v1/key/delete/my-key-*"]
 EOF

Similarly, you can e.g. only allow creating but not deleting secret keys that start with my-key-:

 cat > create-only-policy.toml <<EOF 
 paths = ["/v1/key/create/my-key-*"]
 EOF

2. Isolate Domains

Depending on your setup you may have multiple domains - e.g. customer-A and customer-B. Now, you may want to use one central Vault instance managed by your security team as secure key store. For each customer you set up a new kes server that fetches the secret keys from Vault. However, you want to make sure that the kes server of customer-A can never get access to the key material of customer-B and vice versa.

You can achieve this by assigning each kes server an unique K/V prefix (e.g. customer-A) and grant that kes server only access to to its prefix using Vault's policy system:

path "kv/customer-A/*" {
   capabilities = [ "create", "read", "delete" ]
}

Observe the customer-A prefix. You will need a separate policy and AppRole id/secret pair per domain. So each AppRole role ID must be assigned to the corresponding kv/customer-... policy.

Then you have to tell the kes server to look for secret keys under a prefix. Therefore, you have to adjust the vault section of kes server A's config file:

[keystore.vault]
address = "<vault-address>"
name = "customer-A" # This is the prefix
[keystore.vault.approle]
id     = "<approle-role-id-for-customer-A-policy>"
secret = "<approle-secret-id-for-customer-A-policy>"
retry  = "15s"
[keystore.vault.status]
ping = "10s"

This isolation is separate and independent of Vault Namespaces and can be used on top or instead of namespaces.

Clone this wiki locally