-
Notifications
You must be signed in to change notification settings - Fork 0
kes walkthru (bare metal minio‐kes‐vault)
Allan Roger Reid edited this page Dec 4, 2024
·
5 revisions
See https://github.com/minio/kes/wiki/MinIO-Object-Storage
1.- Create kes-vault, kes-server and kes-minio on same node to share the same network, under lab.min.dev domain, with Enable TLS and SSL Required
ssh -p 20243 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"
Install and validate kes
loginctl enable-linger ubuntu
curl -sSL --tlsv1.2 'https://github.com/minio/kes/releases/latest/download/kes-linux-amd64' -o ./kes
chmod +x ./kes
./kes --version
Output
Version 2024-03-28T12-56-37Z commit=f7a894a79b607ec9d06e1ee7869cd0d736eabfea
Runtime go1.21.8 linux/amd64 compiler=gc
License AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
Copyright 2015-2024 MinIO Inc. https://min.io
ssh -p 20721 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Install Vault
loginctl enable-linger ubuntu
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault
ip -br -4 a
lo UNKNOWN 127.0.0.1/8
eth0@if29 UP 10.214.226.172/24 metric 100
./kes identity new --key vault.key --cert vault.crt --ip 10.214.226.172 kes-vault
Output
Your API key:
kes:v1:AIVqXy/3pX4lfmOt0IP+GoMAQteaVWIEqUDMqIqhwBdF
This is the only time it is shown. Keep it secret and secure!
Your Identity:
14370e7b3fadc5c80b19b3c2358bc97ab912f3654f0e676490c25a67d0a274b7
The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.
The generated TLS private key is stored at: vault.key
The generated TLS certificate is stored at: vault.crt
The identity can be computed again via:
kes identity of kes:v1:AIVqXy/3pX4lfmOt0IP+GoMAQteaVWIEqUDMqIqhwBdF
kes identity of vault.crt
cat vault.crt # on kes-server
vi vault.crt # on kes-vault
cat vault.key # on kes-server
vi vault.key # on kes-vault
vi vault-config.json
# starts a single Vault server instance on port 9020
{
"api_addr": "https://127.0.0.1:9020",
"backend": {
"file": {
"path": "vault/file"
}
},
"default_lease_ttl": "168h",
"max_lease_ttl": "720h",
"listener": {
"tcp": {
"address": "0.0.0.0:9020",
"tls_cert_file": "vault.crt",
"tls_key_file": "vault.key",
"tls_min_version": "tls12"
}
},
"disable_mlock": true
}
sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))
vault server -config vault-config.json > out.log &
tail -f out.log
Output
Log Level:
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: file
Version: Vault v1.15.5, built 2024-01-26T14:53:40Z
Version Sha: 0d8b67ef63815f20421c11fe9152d435af3403e6
==> Vault server started! Log data will stream in below:
==> Vault shutdown triggered
tail: out.log: file truncated
==> Vault server configuration:
Administrative Namespace:
Api Address: https://127.0.0.1:9020
Cgo: disabled
Cluster Address: https://127.0.0.1:9021
Environment Variables: DBUS_SESSION_BUS_ADDRESS, GODEBUG, GOTRACEBACK, HOME, LANG, LESSCLOSE, LESSOPEN, LOGNAME, LS_COLORS, PATH, PWD, SHELL, SHLVL, SSH_CLIENT, SSH_CONNECTION, SSH_TTY, TERM, USER, XDG_DATA_DIRS, XDG_RUNTIME_DIR, XDG_SESSION_CLASS, XDG_SESSION_ID, XDG_SESSION_TYPE, _
Go Version: go1.21.5
Listener 1: tcp (addr: "0.0.0.0:9020", cluster address: "0.0.0.0:9021", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level:
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: file
Version: Vault v1.15.5, built 2024-01-26T14:53:40Z
Version Sha: 0d8b67ef63815f20421c11fe9152d435af3403e6
2024-04-10T20:30:24.335Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2024-04-10T20:30:24.335Z [INFO] incrementing seal generation: generation=1
2024-04-10T20:30:24.346Z [INFO] core: Initializing version history cache for core
2024-04-10T20:30:24.346Z [INFO] events: Starting event system
==> Vault server started! Log data will stream in below:
ssh -p 20721 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
export VAULT_ADDR='https://127.0.0.1:9020'
export VAULT_SKIP_VERIFY=true
vault operator init
Output
Unseal Key 1: 3JR9AKcJG53S6OGhs6g5g67cs4R2rSbb/oApOobwZEp4
Unseal Key 2: RlqgGI+mrUNR7883UzdVsyzT4jeLhRajBTXBb64y+MUb
Unseal Key 3: KfmtXtXagOSCpjDDCAQS+HMBVB/N8idpKeleuUs0YrKd
Unseal Key 4: xO5Gx4Fm+fJpdwXraVNQrbp8csLMWhasUixsZFgpNfGR
Unseal Key 5: /7wDYp7TpeSGz30zv4X+XqUSwtmuJs/xzsEezofh3KBL
Initial Root Token: hvs.FLORrF9CQyb3XeT9w0DZlF1d
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
export VAULT_TOKEN=hvs.FLORrF9CQyb3XeT9w0DZlF1d
vault status
vault operator unseal 3JR9AKcJG53S6OGhs6g5g67cs4R2rSbb/oApOobwZEp4
vault operator unseal RlqgGI+mrUNR7883UzdVsyzT4jeLhRajBTXBb64y+MUb
vault operator unseal KfmtXtXagOSCpjDDCAQS+HMBVB/N8idpKeleuUs0YrKd
vault status
vault secrets enable -version=2 kv
Output
Success! Enabled the kv secrets engine at: kv/
vi kes-policy.hcl
path "kv/data/*" {
capabilities = [ "create", "read" ]
}
path "kv/metadata/*" {
capabilities = [ "list", "delete" ]
}
vi kes-policy-readonly.hcl
path "kv/data/*" {
capabilities = [ "read" ]
}
path "kv/metadata/*" {
capabilities = [ "list" ]
}
path "auth/token/renew-self" {
capabilities = [ "update" ]
}
vault policy write kes-policy kes-policy.hcl
Output
Success! Uploaded policy: kes-policy
vault policy write kes-policy-readonly kes-policy-readonly.hcl
Output
Success! Uploaded policy: kes-policy-readonly
vault auth enable approle
Output
Success! Enabled approle auth method at: approle/
vault write auth/approle/role/kes-server token_num_uses=0 secret_id_num_uses=0 period=5m
Output
Success! Data written to: auth/approle/role/kes-server
vault write auth/approle/role/kes-server-readonly token_num_uses=0 secret_id_num_uses=0 period=5m
Output
Success! Data written to: auth/approle/role/kes-server-readonly
vault write auth/approle/role/kes-server policies=kes-policy
Output
Success! Data written to: auth/approle/role/kes-server
vault write auth/approle/role/kes-server-readonly policies=kes-policy-readonly
Output
Success! Data written to: auth/approle/role/kes-server-readonly
vault read auth/approle/role/kes-server/role-id
Output
Key Value
--- -----
role_id 4dc7b668-25ab-8911-98ec-12fcd746244b
vault read auth/approle/role/kes-server-readonly/role-id
Output
Key Value
--- -----
role_id ddb8bf00-cb6b-78a3-78cb-9a436d7a8f1d
vault write -f auth/approle/role/kes-server/secret-id
Key Value
--- -----
secret_id e1828bff-ace5-adc1-20df-eb325f340f6d
secret_id_accessor cb4be53d-927c-e7d7-28e6-5e0d5aecef43
secret_id_num_uses 0
secret_id_ttl 0s
vault write -f auth/approle/role/kes-server-readonly/secret-id
Key Value
--- -----
secret_id 6fefec65-9ca4-7c8b-302b-e8253b73114a
secret_id_accessor 1b84987f-fbfe-c22a-99e5-6f6944e937e1
secret_id_num_uses 0
secret_id_ttl 0s
ssh -p 20243 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"
ip -br -4 a
./kes identity new --key private.key --cert public.crt --ip "10.214.226.181" kes-server
Output
Your API key:
kes:v1:APkpEgrKSFthYOsdQ0UiBunG7ehooWCA/+M8s7yQPvdf
This is the only time it is shown. Keep it secret and secure!
Your Identity:
2d73e134e1ce10ee9d445eff233a65e4886af0814e8012ab54863c1dc898651e
The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.
The generated TLS private key is stored at: private.key
The generated TLS certificate is stored at: public.crt
The identity can be computed again via:
kes identity of kes:v1:APkpEgrKSFthYOsdQ0UiBunG7ehooWCA/+M8s7yQPvdf
kes identity of public.crt
./kes identity new --key=client.key --cert=client.crt minio
Output
Your API key:
kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP
This is the only time it is shown. Keep it secret and secure!
Your Identity:
67348334baf5c30fb2157dd93f65728714f92eba77fcf2515c1bf062d1140d59
The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.
The generated TLS private key is stored at: client.key
The generated TLS certificate is stored at: client.crt
The identity can be computed again via:
kes identity of kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP
kes identity of client.crt
vi config.yml
cat << EOF > config.yml
address: 0.0.0.0:9073 # Listen on all network interfaces on port 9073
admin:
identity: 2d73e134e1ce10ee9d445eff233a65e4886af0814e8012ab54863c1dc898651e # KES sever is admin
tls:
key: private.key # The KES server TLS private key
cert: public.crt # The KES server TLS certificate
policy:
minio:
allow:
- /v1/key/create/minio-key*
- /v1/key/generate/minio-key*
- /v1/key/decrypt/minio-key*
- /v1/key/list/*
- /v1/key/delete/*
identities:
- 67348334baf5c30fb2157dd93f65728714f92eba77fcf2515c1bf062d1140d59 # Use the identity of your client.crt
cache:
# Cache expiry specifies when cache entries expire.
expiry:
# Period after which any cache entries are discarded.
# It determines how often the KES server has to fetch
# a secret key from the KMS.
#
# If not set, KES will default to an expiry of 5 minutes.
any: 2m0s
# Period after which all unused cache entries are discarded.
# It determines how often "not frequently" used secret keys
# must be fetched from the KMS.
#
# If not set, KES will default to an expiry of 30 seconds.
unused: 20s
# Period after which any cache entries in the offline cache
# are discarded.
# It determines how long the KES server can serve stateless
# requests when the KMS key store has become unavailable -
# for example, due to a network outage.
#
# If not set, KES will disable the offline cache.
#
# Offline caching should only be enabled when trying to
# reduce the impact of the KMS key store being unavailable.
offline: 60s
keystore:
vault:
endpoint: https://10.214.226.172:9020
version: v2 # The K/V engine version - either "v1" or "v2".
approle:
id: "4dc7b668-25ab-8911-98ec-12fcd746244b" # Your AppRole ID
secret: "b1442074-1fab-1620-c010-5f4810825ab6" # Your AppRole Secret
retry: 15s
status:
ping: 10s
tls:
ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates
EOF
./kes server --config config.yml > out.log &
tail -f out.log
Connect to kes-server using apis. Use the minio mTLS identity instead of TLS
export KES_SERVER=https://127.0.0.1:9073
export KES_API_KEY=kes:v1:AB8WJC42ra9iqEaPt0dR8Wk92AC96aCBODEmTU6RRpmP
./kes key ls -k
Key
minio-key-1
ssh -p 20806 [email protected] -o ServerAliveInterval=5 -o "ServerAliveCountMax 100000" -o "StrictHostKeyChecking=off"
loginctl enable-linger ubuntu
export MINIO_KMS_KES_ENDPOINT=https://10.214.226.181:9073
cat client.crt
vi client.crt
cat client.key
vi client.key
export MINIO_KMS_KES_CERT_FILE=client.crt
export MINIO_KMS_KES_KEY_FILE=client.key
export MINIO_KMS_KES_KEY_NAME=minio-key
cat public.crt
vi public.crt
export MINIO_KMS_KES_CAPATH=public.crt
mkdir -p $HOME/.minio/certs
cd $HOME/.minio/certs
wget https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64
chmod +x certgen-linux-amd64
./certgen-linux-amd64 -host "127.0.0.1"
cd $HOME
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin
MINIO_KMS_KES_ENDPOINT=https://10.214.226.181:9073 \
MINIO_KMS_KES_KEY_NAME=minio-key \
MINIO_KMS_KES_CAPATH=public.crt \
MINIO_KMS_KES_CERT_FILE=client.crt \
MINIO_KMS_KES_KEY_FILE=client.key \
CI=on \
minio server /tmp/data --certs-dir ~/.minio/certs --address :9000 --console-address :9090 > out.log &
Access minio console - https://kes-minio.lab.min.dev:9090
In UI - create a bucket, then a key ( > Create Key) mc alias set minio https://kes-minio.lab.min.dev:9000 minioadmin minioadmin
mc admin info minio
mc cp --recursive ~/.minio kes-server/test