Hashicorp Vault Keystore

This guide shows how to setup a KES server that uses Vault's K/V engine as a persistent and secure key store:

Vault Server Setup
KES Server Setup
Recommendations

                         ╔═════════════════════════════════════════════╗
┌────────────┐           ║  ┌────────────┐              ┌───────────┐  ║
│ KES Client ├───────────╫──┤ KES Server ├──────────────┤   Vault   │  ║
└────────────┘           ║  └────────────┘              └───────────┘  ║
                         ╚═════════════════════════════════════════════╝

Vault Server Setup

Generate a TLS certificate for Vault

KES and Vault will exchange sensitive information. In particular, KES will send and receive the secret keys from Vault's HTTP API. Therefore, it is necessary to secure the communication between KES and Vault via TLS. Here, we use self-signed certificates for simplicity. For a production setup we highly recommend certificates signed by CA (e.g. your internal CA or a public CA like Let's Encrypt).
- To generate a new private key for Vault's certificate run the following openssl command:
```
openssl ecparam -genkey -name prime256v1 | openssl ec -out vault-tls.key
```
- Then generate a new TLS certificate for the private/public key pair via:
```
openssl req -new -x509 -days 30 -key vault-tls.key -out vault-tls.cert \
  -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"
```
  Here, we assume that we will run Vault locally as https://127.0.0.1:8200. If you want to run Vault on a different host you may adjust the CN and SAN accordingly.

Download and start a Vault Server

If you haven't already, download the Vault binary. To avoid running Vault as root we must grant the binary the ipc_lock capability such that it can use the mlock syscall:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

Then you can use the following Vault configuration to start a Vault server on https://127.0.0.1:8200:

cat > vault-config.json <<EOF
{
   "api_addr": "https://127.0.0.1:8200",
   "backend": {
     "file": {
       "path": "vault/file"
     }
   },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "vault-tls.cert",
      "tls_key_file": "vault-tls.key",
      "tls_min_version": "tls12"
    }
  }
}
EOF

Note that we run Vault with a file backend. For high-availability in a real production environment you may want to use etcd, consul or Vault with integrated storage instead.

Start the Vault server via:

vault server -config vault-config.json

Initialize and unseal Vault

In a separate terminal window set the VAULT_ADDR env. variable to your Vault server:
```
export VAULT_ADDR='https://127.0.0.1:8200'
```
Further, you may want to run export VAULT_SKIP_VERIFY=true if Vault uses a self-signed TLS certificate. When Vault serves a TLS certificate that has been issued by a CA that is trusted by your machine - e.g. Let's Encrypt - then you don't need to run this command.

Then initialize Vault via:
```
vault operator init
```
Vault will print n (5 by default) unseal key shares of which at least m (3 by default) are required to regenerate the actual unseal key to unseal Vault. Therefore, make sure to remember them. In particular, keep those unseal key shares at a secure and durable location.

You should see some output similar to:
```
   Unseal Key 1: eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
   Unseal Key 2: 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
   Unseal Key 3: cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
   Unseal Key 4: /fTPpec5fWpGqWHK+uhnnTNMQyAbl5alUi4iq2yNgyqj
   Unseal Key 5: UPdDVPto+H6ko+20NKmagK40MOskqOBw4y/S51WpgVy/

   Initial Root Token: s.zaU4Gbcu0Wh46uj2V3VuUde0

   Vault is initialized with 5 key shares and a key threshold of 3. Please securely
   distribute the key shares printed above. When the Vault is re-sealed,
   restarted, or stopped, you must supply at least 3 of these keys to unseal it
   before it can start servicing requests.
```
Now, set the env. variable VAULT_TOKEN to the root token printed by the command before:
```
export VAULT_TOKEN=s.zaU4Gbcu0Wh46uj2V3VuUde0
```
Then use any of the previously generated key shares to unseal Vault.
```
vault operator unseal eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
vault operator unseal 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
vault operator unseal cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
```
Once you have submitted enough valid key shares Vault will become unsealed and able to process requests.
Enable Vault's K/V backend

KES will store the secret keys at the Vault K/V backend.
```
vault secrets enable kv
```
Enable AppRole authentication

KES supports Vault's AppRole authentication mechanism.
```
vault auth enable approle
```
Create an access policy for the K/V engine

KES will use an AppRole ID that has the following policy permissions.
```
cat > kes-policy.hcl <<EOF
path "kv/*" {
     capabilities = [ "create", "read", "delete" ]
}
EOF
```
If you don't want to create or delete keys using the KES server but want to manage them directly using e.g. the Vault CLI (vault kv --help) you may use capabilities = [ "read" ] instead.

Then we upload the policy to Vault:
```
vault policy write kes-policy ./kes-policy.hcl
```
Create an new AppRole ID and bind it to a policy

Now, we need to create a new AppRole ID with the required permissions. The application (i.e. KES server) will authenticate to Vault via the AppRole role ID and secret ID and is only allowed to perform operations granted by the kes-policy.hcl policy. So, we first create a new role for our KES server:
```
vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m
```
Then we bind the kes-policy policy to the role:
```
vault write auth/approle/role/kes-role policies=kes-policy
```
Finally, we request an AppRole role ID and secret ID from Vault.
First, the role ID:
```
vault read auth/approle/role/kes-role/role-id 
```
Then, the secret ID:
```
vault write -f auth/approle/role/kes-role/secret-id
```
We are only interested in the secret_id - not in the secret_id_accessor.

KES Server setup

Similar to Vault, KES can use the mlock syscall on linux systems to prevent the OS from writing in-memory data to disk (swapping). Therefore, you should give the KES executable the ability to use the mlock syscall without running the process as root. To do so run:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

First, we need to generate a TLS private key and certificate for our KES server. A KES server can only be run with TLS - since secure-by-default. Here we use self-signed certificates for simplicity. For a production setup we highly recommend to use a certificate signed by CA (e.g. your internal CA or a public CA like Let's Encrypt)

Generate a TLS private key and certificate for the KES server.
The following command will generate a new TLS private key server.key and a X.509 certificate server.cert that is self-signed and issued for the IP 127.0.0.1 and DNS name localhost (as SAN). You may want to customize the command to match your setup.
```
kes tool identity new --server --key server.key --cert server.cert --ip "127.0.0.1" --dns localhost
```
Any other tooling for X.509 certificate generation works as well. For example, you could use openssl:
```
$ openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key

$ openssl req -new -x509 -days 30 -key server.key -out server.cert \
   -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"
```
Then, create private key and certificate for your application:
```
kes tool identity new --key=app.key --cert=app.cert app
```
You can compute the app identity via:
```
kes tool identity of app.cert
```

Now we have defined all entities in our demo setup. Let's wire everything together by creating the config file server-config.yml:

address: 0.0.0.0:7373
root:    disabled  # We disable the root identity since we don't need it in this guide 

tls:
  key:  server.key
  cert: server.cert

policy:    
  my-app: 
    paths:
    - /v1/key/create/my-app*
    - /v1/key/generate/my-app*
    - /v1/key/decrypt/my-app*
    identities:
    - ${APP_IDENTITY}

keys:
  vault:
    endpoint: https://127.0.0.1:8200
    approle:
      id:     "" # Your AppRole ID
      secret: "" # Your AppRole Secret ID
      retry:  15s
    status:
      ping: 10s
    tls:
      ca: vault-tls.crt # Since we use self-signed certificates

Please use your AppRole credentials.

Finally we can start a KES server in a new window/tab:
```
export APP_IDENTITY=$(kes tool identity of app.cert)

kes server --mlock --config=server-config.yml --auth=off
```
--auth=off is required since our root.cert and app.cert certificates are self-signed.
The --mlock option is currently only available on Linux.
In the previous window/tab we now can connect to the server by:
```
export KES_CLIENT_CERT=app.cert
export KES_CLIENT_KEY=app.key
kes key create -k my-app-key
```
-k is required because we use self-signed certificates

Now, you should see a secret key at the Vault K/V backend. You can use the Vault CLI to inspect the K/V backend:
```
vault kv list kv/
```

Finally, we can derive and decrypt data keys from the previously created app-key:

kes key derive -k my-app-key
{
  plaintext : ...
  ciphertext: ...
}

kes key decrypt -k my-app-key <base64-ciphertext>

Recommendations

Here we show some additional configuration steps that can solve specific problems.

1. Control who can create secret keys.

In general, the KES server as well as Vault can create and delete secret keys. The KES server exposes the /v1/key/create and v1/key/decrypt REST API. Vault's K/V store exposes an API for creating and deleting secrets.

Now, you can control which one can create (or delete) secret keys using the kes-policy.hcl. If you want to e.g. grant the KES server the permission to create and delete keys from Vault's K/V store then use the following policy:

path "kv/*" {
   capabilities = [ "create", "read", "delete" ]
}

However, you can also grant only the permission to create but not to delete keys using:

path "kv/*" {
   capabilities = [ "create", "read" ]
}

You should not grant the KES server the permission to "update" secret keys. It's not a security issue if you do but it's a) not necessary and b) it can lead to data corruption in distributed systems.

This policy only specifies whether the KES server, in general, can create or delete K/V entries. However, without a policy that explicitly grants access to the /v1/key/create or /v1/key/delete API, only the root identity can perform those operations. If a KES client should be able to create and delete keys that match my-app*you can use the following policy snippet in the policy section of your config:

policy:    
  my-app: 
    paths:
    - /v1/key/create/my-app*
    - /v1/key/delete/my-app*

Similarly, you can e.g. only allow creating but not deleting secret keys that match my-app*:

policy:    
  my-app: 
    paths:
    - /v1/key/create/my-app*

2. Isolate Domains

Depending on your setup you may have multiple domains - e.g. customer-A and customer-B. Now, you may want to use one central Vault instance managed by your security team as secure key store. For each customer you set up a new KES server that fetches the secret keys from Vault. However, you want to make sure that the KES server of customer-A can never get access to the key material of customer-B and vice versa.

You can achieve this by assigning each KES server an unique K/V prefix (e.g. customer-A) and grant that KES server only access to to its prefix using Vault's policy system:

path "kv/customer-A/*" {
   capabilities = [ "create", "read", "delete" ]
}

Observe the customer-A prefix. You will need a separate policy and AppRole id/secret pair per domain. So each AppRole role ID must be assigned to the corresponding kv/customer-... policy.

Then you have to tell the KES server to look for secret keys under a prefix. Therefore, you have to adjust the vault section of KES server A's config file:

keys:
  vault:
    endpoint: https://127.0.0.1:8200
    prefix:   customer-A
    approle:
      id:     "" # Your AppRole ID
      secret: "" # Your AppRole Secret ID
      retry:  15s
    status:
      ping: 10s

This isolation is separate and independent of Vault Namespaces and can be used on top or instead of namespaces.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hashicorp Vault Keystore

Vault Server Setup

Generate a TLS certificate for Vault

Download and start a Vault Server

Initialize and unseal Vault

Enable Vault's K/V backend

Enable AppRole authentication

Create an access policy for the K/V engine

Create an new AppRole ID and bind it to a policy

KES Server setup

Recommendations

1. Control who can create secret keys.

2. Isolate Domains

Clone this wiki locally