Skip to content

kes walkthru (bare metal minio‐kes‐vault)

Allan Roger Reid edited this page Dec 4, 2024 · 5 revisions

See https://github.com/minio/kes/wiki/MinIO-Object-Storage

1.- Create kes-vault, kes-server and kes-minio on same node to share the same network, under lab.min.dev domain, with Enable TLS and SSL Required

2.- Connect to kes-server

ssh -p 20243 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"

Install and validate kes

loginctl enable-linger ubuntu 
curl -sSL --tlsv1.2 'https://github.com/minio/kes/releases/latest/download/kes-linux-amd64' -o ./kes
chmod +x ./kes
./kes --version

Output

Version    2024-03-28T12-56-37Z   commit=f7a894a79b607ec9d06e1ee7869cd0d736eabfea
Runtime    go1.21.8 linux/amd64   compiler=gc
License    AGPLv3                 https://www.gnu.org/licenses/agpl-3.0.html
Copyright  2015-2024 MinIO Inc.   https://min.io

3.- Connect to kes-vault

ssh -p 20721 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Install Vault

loginctl enable-linger ubuntu 
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault
ip -br -4 a
lo               UNKNOWN        127.0.0.1/8 
eth0@if29        UP             10.214.226.172/24 metric 100 

4.- Return to kes-server tab

Generate Vault Private Key & Certificate

./kes identity new --key vault.key --cert vault.crt --ip 10.214.226.172 kes-vault

Output

Your API key:

   kes:v1:AIVqXy/3pX4lfmOt0IP+GoMAQteaVWIEqUDMqIqhwBdF

This is the only time it is shown. Keep it secret and secure!

Your Identity:

   14370e7b3fadc5c80b19b3c2358bc97ab912f3654f0e676490c25a67d0a274b7

The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.

The generated TLS private key is stored at: vault.key
The generated TLS certificate is stored at: vault.crt

The identity can be computed again via:

    kes identity of kes:v1:AIVqXy/3pX4lfmOt0IP+GoMAQteaVWIEqUDMqIqhwBdF
    kes identity of vault.crt

5.- Return to kes-vault tab

Copy vault.crt and vault.key from kes-server to kes-vault

cat vault.crt # on kes-server
vi vault.crt # on kes-vault

cat vault.key # on kes-server
vi vault.key # on kes-vault

Configure Vault Server

vi vault-config.json

# starts a single Vault server instance on port 9020
{
  "api_addr": "https://127.0.0.1:9020",
  "backend": {
    "file": {
      "path": "vault/file"
    }
  },

  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",

  "listener": {
    "tcp": {
      "address": "0.0.0.0:9020",
      "tls_cert_file": "vault.crt",
      "tls_key_file": "vault.key",
      "tls_min_version": "tls12"
    }
  },
  "disable_mlock": true
}

Start vault server

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))
vault server -config vault-config.json > out.log &
tail -f out.log

Output

               Log Level: 
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.15.5, built 2024-01-26T14:53:40Z
             Version Sha: 0d8b67ef63815f20421c11fe9152d435af3403e6

==> Vault server started! Log data will stream in below:

==> Vault shutdown triggered
tail: out.log: file truncated
==> Vault server configuration:

Administrative Namespace: 
             Api Address: https://127.0.0.1:9020
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:9021
   Environment Variables: DBUS_SESSION_BUS_ADDRESS, GODEBUG, GOTRACEBACK, HOME, LANG, LESSCLOSE, LESSOPEN, LOGNAME, LS_COLORS, PATH, PWD, SHELL, SHLVL, SSH_CLIENT, SSH_CONNECTION, SSH_TTY, TERM, USER, XDG_DATA_DIRS, XDG_RUNTIME_DIR, XDG_SESSION_CLASS, XDG_SESSION_ID, XDG_SESSION_TYPE, _
              Go Version: go1.21.5
              Listener 1: tcp (addr: "0.0.0.0:9020", cluster address: "0.0.0.0:9021", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: 
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.15.5, built 2024-01-26T14:53:40Z
             Version Sha: 0d8b67ef63815f20421c11fe9152d435af3403e6

2024-04-10T20:30:24.335Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2024-04-10T20:30:24.335Z [INFO]  incrementing seal generation: generation=1
2024-04-10T20:30:24.346Z [INFO]  core: Initializing version history cache for core
2024-04-10T20:30:24.346Z [INFO]  events: Starting event system
==> Vault server started! Log data will stream in below:

6.- In new terminal session to kes-vault

ssh -p 20721 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Set VAULT_ADDR endpoint

export VAULT_ADDR='https://127.0.0.1:9020'
export VAULT_SKIP_VERIFY=true

Initialize Vault Server

vault operator init

Output

Unseal Key 1: 3JR9AKcJG53S6OGhs6g5g67cs4R2rSbb/oApOobwZEp4
Unseal Key 2: RlqgGI+mrUNR7883UzdVsyzT4jeLhRajBTXBb64y+MUb
Unseal Key 3: KfmtXtXagOSCpjDDCAQS+HMBVB/N8idpKeleuUs0YrKd
Unseal Key 4: xO5Gx4Fm+fJpdwXraVNQrbp8csLMWhasUixsZFgpNfGR
Unseal Key 5: /7wDYp7TpeSGz30zv4X+XqUSwtmuJs/xzsEezofh3KBL

Initial Root Token: hvs.FLORrF9CQyb3XeT9w0DZlF1d

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Set VAULT_TOKEN

export VAULT_TOKEN=hvs.FLORrF9CQyb3XeT9w0DZlF1d

Unseal Vault Server

vault status
vault operator unseal 3JR9AKcJG53S6OGhs6g5g67cs4R2rSbb/oApOobwZEp4
vault operator unseal RlqgGI+mrUNR7883UzdVsyzT4jeLhRajBTXBb64y+MUb
vault operator unseal KfmtXtXagOSCpjDDCAQS+HMBVB/N8idpKeleuUs0YrKd
vault status

Enable K/V Backend

vault secrets enable -version=2 kv

Output

Success! Enabled the kv secrets engine at: kv/

Create Vault Policy

vi kes-policy.hcl
path "kv/data/*" {
   capabilities = [ "create", "read" ]
}
path "kv/metadata/*" {
   capabilities = [ "list", "delete" ]       
}
vi kes-policy-readonly.hcl
path "kv/data/*" {
   capabilities = [ "read" ]
}
path "kv/metadata/*" {
   capabilities = [ "list" ]       
}
path "auth/token/renew-self" {
   capabilities = [ "update" ]       
}

Then,

vault policy write kes-policy kes-policy.hcl

Output

Success! Uploaded policy: kes-policy
vault policy write kes-policy-readonly kes-policy-readonly.hcl

Output

Success! Uploaded policy: kes-policy-readonly

Enable AppRole Authentication

vault auth enable approle

Output

Success! Enabled approle auth method at: approle/

Create KES Role

vault write auth/approle/role/kes-server token_num_uses=0  secret_id_num_uses=0  period=5m

Output

Success! Data written to: auth/approle/role/kes-server
vault write auth/approle/role/kes-server-readonly token_num_uses=0  secret_id_num_uses=0  period=5m

Output

Success! Data written to: auth/approle/role/kes-server-readonly

Bind Policy to Role

vault write auth/approle/role/kes-server policies=kes-policy

Output

Success! Data written to: auth/approle/role/kes-server
vault write auth/approle/role/kes-server-readonly policies=kes-policy-readonly

Output

Success! Data written to: auth/approle/role/kes-server-readonly

Generate AppRole ID

vault read auth/approle/role/kes-server/role-id 

Output

Key        Value
---        -----
role_id    4dc7b668-25ab-8911-98ec-12fcd746244b
vault read auth/approle/role/kes-server-readonly/role-id 

Output

Key        Value
---        -----
role_id    ddb8bf00-cb6b-78a3-78cb-9a436d7a8f1d

Generate AppRole Secret

vault write -f auth/approle/role/kes-server/secret-id 
Key                   Value
---                   -----
secret_id             e1828bff-ace5-adc1-20df-eb325f340f6d
secret_id_accessor    cb4be53d-927c-e7d7-28e6-5e0d5aecef43
secret_id_num_uses    0
secret_id_ttl         0s
vault write -f auth/approle/role/kes-server-readonly/secret-id 
Key                   Value
---                   -----
secret_id             6fefec65-9ca4-7c8b-302b-e8253b73114a
secret_id_accessor    1b84987f-fbfe-c22a-99e5-6f6944e937e1
secret_id_num_uses    0
secret_id_ttl         0s

7.- In new terminal session to kes-server

ssh -p 20243 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"

Generate KES Server Private Key & Certificate with its own IP

ip -br -4 a
./kes identity new --key private.key --cert public.crt --ip "10.214.226.181" kes-server

Output

Your API key:

   kes:v1:APkpEgrKSFthYOsdQ0UiBunG7ehooWCA/+M8s7yQPvdf

This is the only time it is shown. Keep it secret and secure!

Your Identity:

   2d73e134e1ce10ee9d445eff233a65e4886af0814e8012ab54863c1dc898651e

The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.

The generated TLS private key is stored at: private.key
The generated TLS certificate is stored at: public.crt

The identity can be computed again via:

    kes identity of kes:v1:APkpEgrKSFthYOsdQ0UiBunG7ehooWCA/+M8s7yQPvdf
    kes identity of public.crt

Generate Client Credentials

./kes identity new --key=client.key --cert=client.crt minio

Output

Your API key:

   kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP

This is the only time it is shown. Keep it secret and secure!

Your Identity:

   67348334baf5c30fb2157dd93f65728714f92eba77fcf2515c1bf062d1140d59

The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.

The generated TLS private key is stored at: client.key
The generated TLS certificate is stored at: client.crt

The identity can be computed again via:

    kes identity of kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP
    kes identity of client.crt

Configure KES Server (make minio client and kes admin)

vi config.yml
cat << EOF > config.yml
address: 0.0.0.0:9073 # Listen on all network interfaces on port 9073

admin:
  identity: 2d73e134e1ce10ee9d445eff233a65e4886af0814e8012ab54863c1dc898651e # KES sever is admin
   
tls:
  key: private.key    # The KES server TLS private key
  cert: public.crt    # The KES server TLS certificate
   
policy:
  minio: 
    allow:
    - /v1/key/create/minio-key*
    - /v1/key/generate/minio-key*
    - /v1/key/decrypt/minio-key*
    - /v1/key/list/*
    - /v1/key/delete/*
    identities:
    - 67348334baf5c30fb2157dd93f65728714f92eba77fcf2515c1bf062d1140d59 # Use the identity of your client.crt

cache:
  # Cache expiry specifies when cache entries expire.
  expiry:
    # Period after which any cache entries are discarded.
    # It determines how often the KES server has to fetch
    # a secret key from the KMS.
    #
    # If not set, KES will default to an expiry of 5 minutes.
    any: 2m0s
    # Period after which all unused cache entries are discarded.
    # It determines how often "not frequently" used secret keys
    # must be fetched from the KMS.
    #
    # If not set, KES will default to an expiry of 30 seconds.
    unused: 20s
    # Period after which any cache entries in the offline cache
    # are discarded.
    # It determines how long the KES server can serve stateless
    # requests when the KMS key store has become unavailable -
    # for example, due to a network outage.
    #
    # If not set, KES will disable the offline cache.
    #
    # Offline caching should only be enabled when trying to
    # reduce the impact of the KMS key store being unavailable.
    offline: 60s

keystore:
   vault:
     endpoint: https://10.214.226.172:9020
     version:  v2 # The K/V engine version - either "v1" or "v2".
     approle:
       id:     "4dc7b668-25ab-8911-98ec-12fcd746244b" # Your AppRole ID
       secret: "b1442074-1fab-1620-c010-5f4810825ab6" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates
EOF

Start KES Server

./kes server --config config.yml > out.log &
tail -f out.log

Validate

Connect to kes-server using apis. Use the minio mTLS identity instead of TLS

export KES_SERVER=https://127.0.0.1:9073
export KES_API_KEY=kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP
  
./kes key ls -k
Key
minio-key-1

8.- Connect to kes-minio

ssh -p 20806 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"
loginctl enable-linger ubuntu 

Set MINIO_KMS_KES_ENDPOINT

export MINIO_KMS_KES_ENDPOINT=https://10.214.226.181:9073

Set MinIO Client Credentials. Copy from kes-server

cat client.crt
vi client.crt
cat client.key
vi client.key
export MINIO_KMS_KES_CERT_FILE=client.crt
export MINIO_KMS_KES_KEY_FILE=client.key

Set MinIO Default Key

export MINIO_KMS_KES_KEY_NAME=minio-key

Trust the KES Server Certificate. Copy from kes-server

cat public.crt
vi public.crt
export MINIO_KMS_KES_CAPATH=public.crt

Start MinIO Server

Make certs
mkdir -p $HOME/.minio/certs
cd $HOME/.minio/certs
wget https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64
chmod +x certgen-linux-amd64
./certgen-linux-amd64 -host "127.0.0.1"
cd $HOME 
Install and run minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc 
sudo mv mc /usr/local/bin

MINIO_KMS_KES_ENDPOINT=https://10.214.226.181:9073 \
MINIO_KMS_KES_KEY_NAME=minio-key \
MINIO_KMS_KES_CAPATH=public.crt \
MINIO_KMS_KES_CERT_FILE=client.crt \
MINIO_KMS_KES_KEY_FILE=client.key \
CI=on \
minio server /tmp/data --certs-dir ~/.minio/certs --address :9000 --console-address :9090 > out.log &

Access minio console - https://kes-minio.lab.min.dev:9090

In UI - create a bucket, then a key ( > Create Key) mc alias set minio https://kes-minio.lab.min.dev:9000 minioadmin minioadmin

mc admin info minio
mc cp --recursive ~/.minio kes-server/test
Clone this wiki locally