Install stateful KES in k8s
Install stateful KES in k8s
- https://github.com/cniackz/public/wiki/Install-stateful-KES
- https://github.com/minio/operator/blob/master/examples/kustomization/tenant-env-encryption/tenant.yaml
- Delete previous cluster
kind delete clusters kind- Create new cluster
kind create cluster --config ~/operator/testing/kind-config.yaml- Deploy Operator:
kubectl apply -k github.com/minio/operator/- Deploy Tenant
kubectl apply -k ~/operator/examples/kustomization/tenant-lite-
Create and configure a pod for kes:
- Create an Ubuntu Pod for KES inside the tenant namespace:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: ubuntu
namespace: tenant-lite
labels:
app: ubuntu
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "604800"
imagePullPolicy: IfNotPresent
name: ubuntu
restartPolicy: Always
EOF- Install KES and
mcin the Ubuntu Pod:
apt update
apt install wget
apt install vim
wget https://github.com/minio/kes/releases/latest/download/kes-linux-amd64
mv kes-linux-amd64 kes
chmod +x kes
mv kes /usr/local/bin/kes
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc
- Create a persistent directory for KES and its configuration file inside the ubuntu pod where kes is located:
rm -rf ~/kes
mkdir ~/kes
cd ~/kes
touch init.yml- Create identities in ubuntu pod:
cd ~/kes
kes identity new --key sys-admin.key --cert sys-admin.crt kes-sys-admin
kes identity new --key minio-admin.key --cert minio-admin.crt minio-admin
kes identity new --key minio.key --cert minio.crt minio
kes identity new --ip "10.244.2.7" localhost # will generate private.key and public.crt
|
|____ IP Address of the Ubuntu Pod.Expected 4 identities:
root@ubuntu:/# cd ~/kes
root@ubuntu:~/kes# ls
data init.yml minio-admin.crt minio-admin.key minio.crt minio.key private.key public.crt sys-admin.crt sys-admin.key
| | | | | |
| | |__________|___ minio |________________|___ kes-sys-admin
| |
|_______________|___ minio-admin
root@ubuntu:~/kes# - Create KES unseal key:
cat /dev/urandom | head -c 32 | base64 # put the result in the .bashrc
vi ~/.bashrc
export KES_UNSEAL_KEY=<VALUE-FROM-ABOVE-COMMAND>
source ~/.bashrc
echo $KES_UNSEAL_KEY # it should print the value- Edit/Create KES config file:
cd ~/kes
echo "version: v1" > ~/kes/init.yml
echo "address: 0.0.0.0:7373" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "tls:" >> ~/kes/init.yml
echo " key: private.key" >> ~/kes/init.yml
echo " cert: public.crt" >> ~/kes/init.yml
echo " client:" >> ~/kes/init.yml
echo " verify_cert: false" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "system:" >> ~/kes/init.yml
echo " admin:" >> ~/kes/init.yml
echo " identity: $(kes identity of sys-admin.crt)" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "unseal:" >> ~/kes/init.yml
echo " environment:" >> ~/kes/init.yml
echo " name: KES_UNSEAL_KEY" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "enclave:" >> ~/kes/init.yml
echo " default:" >> ~/kes/init.yml
echo " admin:" >> ~/kes/init.yml
echo " identity: $(kes identity of minio-admin.crt)" >> ~/kes/init.yml
echo " policy:" >> ~/kes/init.yml
echo " minio:" >> ~/kes/init.yml
echo " allow:" >> ~/kes/init.yml
echo " - /v1/api" >> ~/kes/init.yml
echo " - /v1/log/audit" >> ~/kes/init.yml
echo " - /v1/log/error" >> ~/kes/init.yml
echo " - /v1/key/create/*" >> ~/kes/init.yml
echo " - /v1/key/generate/*" >> ~/kes/init.yml
echo " - /v1/key/decrypt/*" >> ~/kes/init.yml
echo " - /v1/key/bulk/decrypt/*" >> ~/kes/init.yml- Initialize KES deployment
cd ~/kes # where init.yml is saved
kes init --config init.yml ~/kes/data- Start KES server
kes server ~/kes/data- In Ubuntu Pod Terminal where KES is located: Assign MinIO identity to MinIO policy:
cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)
Expected:
root@ubuntu:/# cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes# printenv | grep KES_SERVER
KES_SERVER=https://127.0.0.1:7373
root@ubuntu:~/kes#
- previous steps:
root@ubuntu:~/kes# cat ~/kes/minio.key
-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEICp6YACcpHa7AXRRxN+T+8JpgU35AEZ05rMcX+1nc3O4
-----END PRIVATE KEY-----
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes#
root@ubuntu:~/kes# ~/kes/minio.crt
bash: /root/kes/minio.crt: Permission denied
root@ubuntu:~/kes# cat ~/kes/minio.crt
-----BEGIN CERTIFICATE-----
MIIBHTCB0KADAgECAhEA3H4g9US91T1xyKYGjI+WUDAFBgMrZXAwEDEOMAwGA1UE
AxMFbWluaW8wHhcNMjMwMTMxMTkyNTI0WhcNMjMwMzAyMTkyNTI0WjAQMQ4wDAYD
VQQDEwVtaW5pbzAqMAUGAytlcAMhAGWmca0XXen/PwJRwd8yQAq26hE+Jmsw3oGq
sJgePljroz8wPTAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwBQYDK2VwA0EA8/OSVV54qLd3MbUatntJ
Rhe/+3FGE15nCqcjQHKzVWtYHMmxw5dFJKokJljZ6F0BfeviTCWroWQbdnRqD4Hr
AQ==
-----END CERTIFICATE-----
root@ubuntu:~/kes#- Created the secrets:
# /Users/cniackz/minio/private.key is ~/kes/minio.key
# /Users/cniackz/minio/public.crt is ~/kes/minio.crt
# These two files below comes from this line above: `kes identity new --key minio.key --cert minio.crt minio`
kubectl create secret generic kes-minio -n tenant-lite --from-file=~/kes/minio.key --from-file=~/kes/minio.crt
# /Users/cniackz/minio/private.key is ~/kes/minio.key
# /Users/cniackz/minio/public.crt is ~/kes/public.crt
# I copied and pasted the private.key and public.crt content from the ubuntu pod down to my computer,
# in order to apply below command from my computer:
# ~/kes/private.key <---- This is coming from the ubuntu pod at the end
# and the command that generates these files is kes identity new --ip "someipaddress" localhost
# please download the files to your computer or have kubectl access from the pod or other access.
kubectl create secret generic kes-minio-public -n tenant-lite --from-file=~/kes/private.key --from-file=~/kes/public.crtk edit tenant -n tenant-lite
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
name: storage
namespace: minio-tenant
spec:
# externalClientCertSecrets is to share the secret with the MinIO Pods:
# Under: /tmp/certs/client-0 You will find:
# client.crt and client.key
# And we can use these files to setup KES in k8s
externalClientCertSecrets:
- name: kes-minio
type: Opaque
- name: kes-minio-public
type: Opaque
env:
# Set MINIO_KMS_KES_ENDPOINT
# It is the IP of the Ubuntu Pod.
- name: MINIO_KMS_KES_ENDPOINT
value: "https://10.244.2.7:7373"
# Set MinIO Client Credentials, it comes from kes-minio secret
- name: MINIO_KMS_KES_CERT_FILE
value: "/tmp/certs/client-0/client.crt"
# Set MinIO Client Credentials, it comes from kes-minio secret
- name: MINIO_KMS_KES_KEY_FILE
value: "/tmp/certs/client-0/client.key"
# Set MinIO Default Key
- name: MINIO_KMS_KES_KEY_NAME
value: "minio-default-key"
# Trust the KES Server Certificate, it comes from kes-minio-public secret
- name: MINIO_KMS_KES_CAPATH
value: "/tmp/certs/client-1/client.crt"
# Root User
- name: MINIO_ROOT_USER
value: minio
# ROOT Password:
- name: MINIO_ROOT_PASSWORD
value: minio123
mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc mb myminio/my-bucket # create new bucket
mc admin kms key create myminio minio-my-bucket # create key
mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucketroot@ubuntu:/# mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
Added `myminio` successfully.
root@ubuntu:/# mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc: <ERROR> Unable to validate target `myminio/my-bucket`. Bucket `my-bucket` does not exist.
root@ubuntu:/# mc mb myminio/my-bucket # create new bucket
Bucket created successfully `myminio/my-bucket`.
root@ubuntu:/# mc admin kms key create myminio minio-my-bucket # create key
Created master key `minio-my-bucket` successfully
root@ubuntu:/# mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket
Auto encryption configuration has been set successfully for myminio/my-bucket
root@ubuntu:/# - Question:
KES is saving the keys in local memory right?, we are not using KMS at all, correct? Since the idea is to avoid KMS/Vault.
- Answer:
KES is saving keys, identities and policies on a local drive - like MinIO stores objects in local (and remote) drives.
Therefore, it is stateful. All state and persistence is maintained by KES itself. In case of stateless KES, KES relies on an external system - e.g. Vault - to
persist keys.
- Question:
What are the advantages or disadvantages of using stateful kes in k8s? or in general...
- Answer:
One major advantage of stateful KES is: It simplifies the deployment setup and reduces external dependencies.
In case of stateless KES, there must be some external KMS system. For example, you need to setup Vault first before you can deploy KES.
As a user/customer of MinIO you have to:
1. Setup MinIO (and pay for MinIO's SubNet)
2. Talk to an external KMS vendor and purchase their product as well.
You have two entities that have to work together and you probably pay both.
By using stateful KES, you can rule out the external KMS vendor since you get the KMS for free.
Whether stateful KES is beneficial for a customer depends upon whether they already have a (centrally managed) KMS solution.
If they do, then it may be easier to simply connect (stateless) KES to their KMS. Maybe their KMS powers other applications as well.
However, if they don't have such a solution, yet, or if it is not usable / isolated and they would have to purchase a KMS solution
in any case, then stateful KES can simplify their lives.
- Question:
Why do we need to create KES unseal key? what is this for?
- Answer:
As mentioned, stateful KES handles all data itself. Since it stores all crypto. keys, policies etc. locally it has to encrypt all data it writes to disk.
The unseal key is basically the key to unlock this encrypted storage.
Currently, KES supports a very simply unseal mechanism: There is a 256 bit encryption key which you provide via env. var.
However, KES is designed to support more complex unseal mechanisms in the future (there is a generic internal implementation).
For example, KES could implement unsealing via a hardware module - like TPM or HSM - or by talking to an external auth service.
The unsealing concept is not unique to KES. Any reasonable KMS implementation has to provide such a feature.
Hashicorp Vault defaults to an unseal mechanism called Secret Splitting. Instead of one unseal key you have N (configurable)
unseal keys out of which you need M (configurable) unseal keys to unlock Vaults encrypted storage. It is a threshold mechansim
where e.g. 3 out of 5 unseal keys are required to unlock Vault.
KES could implement such a scheme too, but we decided not to for now.
When the unseal mechanism gets more complex then the initial learning curve for deploying gets significantly steeper. Therefore,
we went for the most basic unseal mechanism.
You specify the unseal mechanism in the kes init config file. Right now there is just environment but we can add more anytime in the future.
unseal:
environment:
name: KES_UNSEAL_KEY