Skip to content

Install stateful KES in k8s

Jump to bottom
Cesar Celis Hernandez edited this page Jan 31, 2023 · 36 revisions

Objective:

Install stateful KES in k8s

Diagram:

Screen Shot 2022-10-05 at 2 39 27 PM

Inspired from:

Steps:

  • Delete previous cluster
kind delete clusters kind
  • Create new cluster
kind create cluster --config ~/operator/testing/kind-config.yaml
  • Deploy Operator:
kubectl apply -k github.com/minio/operator/
  • Deploy Tenant
kubectl apply -k ~/operator/examples/kustomization/tenant-lite

  • Create and configure a pod for kes:

    • Create an Ubuntu Pod for KES inside the tenant namespace:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  namespace: tenant-lite
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always
EOF
  • Install KES and mc in the Ubuntu Pod:
apt update
apt install wget
apt install vim
wget https://github.com/minio/kes/releases/latest/download/kes-linux-amd64
mv kes-linux-amd64 kes
chmod +x kes
mv kes /usr/local/bin/kes
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc
  • Create a persistent directory for KES and its configuration file inside the ubuntu pod where kes is located:
rm -rf ~/kes
mkdir ~/kes
cd ~/kes
touch init.yml
  • Create identities in ubuntu pod:
cd ~/kes
kes identity new --key sys-admin.key --cert sys-admin.crt kes-sys-admin
kes identity new --key minio-admin.key --cert minio-admin.crt minio-admin
kes identity new --key minio.key --cert minio.crt minio
kes identity new --ip "10.244.2.7" localhost # will generate private.key and public.crt
                           |
                           |____ IP Address of the Ubuntu Pod.

Expected 4 identities:

root@ubuntu:/# cd ~/kes
root@ubuntu:~/kes# ls
data  init.yml  minio-admin.crt  minio-admin.key  minio.crt  minio.key  private.key  public.crt  sys-admin.crt  sys-admin.key
                      |               |              |          |                                    |                |
                      |               |              |__________|___ minio                           |________________|___ kes-sys-admin
                      |               |
                      |_______________|___ minio-admin
root@ubuntu:~/kes#
  • Create KES unseal key:
cat /dev/urandom | head -c 32 | base64 # put the result in the .bashrc
vi ~/.bashrc
export KES_UNSEAL_KEY=<VALUE-FROM-ABOVE-COMMAND>
source ~/.bashrc
echo $KES_UNSEAL_KEY # it should print the value
  • Edit/Create KES config file:
cd ~/kes
echo "version: v1" > ~/kes/init.yml
echo "address: 0.0.0.0:7373" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "tls:" >> ~/kes/init.yml
echo "  key: private.key" >> ~/kes/init.yml
echo "  cert: public.crt" >> ~/kes/init.yml
echo "  client:" >> ~/kes/init.yml
echo "    verify_cert: false" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "system:" >> ~/kes/init.yml
echo "  admin:" >> ~/kes/init.yml
echo "    identity: $(kes identity of sys-admin.crt)" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "unseal:" >> ~/kes/init.yml
echo "  environment:" >> ~/kes/init.yml
echo "    name: KES_UNSEAL_KEY" >> ~/kes/init.yml
echo "" >> ~/kes/init.yml
echo "enclave:" >> ~/kes/init.yml
echo "  default:" >> ~/kes/init.yml
echo "    admin:" >> ~/kes/init.yml
echo "      identity: $(kes identity of minio-admin.crt)" >> ~/kes/init.yml
echo "    policy:" >> ~/kes/init.yml
echo "      minio:" >> ~/kes/init.yml
echo "        allow:" >> ~/kes/init.yml
echo "        - /v1/api" >> ~/kes/init.yml
echo "        - /v1/log/audit" >> ~/kes/init.yml
echo "        - /v1/log/error" >> ~/kes/init.yml
echo "        - /v1/key/create/*" >> ~/kes/init.yml
echo "        - /v1/key/generate/*" >> ~/kes/init.yml
echo "        - /v1/key/decrypt/*" >> ~/kes/init.yml
echo "        - /v1/key/bulk/decrypt/*" >> ~/kes/init.yml
  • Initialize KES deployment
cd ~/kes # where init.yml is saved
kes init --config init.yml ~/kes/data

Expected is:

root@ubuntu:~/kes# cd ~/kes # where init.yml is saved
kes init --config init.yml ~/kes/data
TLS:
  · Private Key:  private.key
  · Certificate:  public.crt

System:
  · Identity:  1a65f6f86c3268b30528fe4aab88fc6994730346e1c1863052fa3fa192d77d3e

Unseal:
  · Environment:  KES_UNSEAL_KEY

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                        Initialized KES v0.22.3 in /root/kes/data                                                        │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@ubuntu:~/kes#
  • Start KES server
kes server ~/kes/data

Expected is:

root@ubuntu:~/kes# kes server ~/kes/data
Copyright  MinIO, Inc.  https://min.io
License    GNU AGPLv3   https://www.gnu.org/licenses/agpl-3.0.html
Version    v0.22.3      linux/amd64

Endpoints  https://127.0.0.1:7373
           https://10.244.4.2:7373

mTLS       skip verify  Client certificates are not verified
Mem Lock   off          Failed to lock RAM pages. Consider granting CAP_IPC_LOCK
  • In Ubuntu Pod Terminal where KES is located: Assign MinIO identity to MinIO policy:
cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)

Expected:

root@ubuntu:/# cd ~/kes
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=minio-admin.key
export KES_CLIENT_CERT=minio-admin.crt
kes policy assign -k minio $(kes identity of minio.crt)
root@ubuntu:~/kes# 
root@ubuntu:~/kes# 
root@ubuntu:~/kes# 
root@ubuntu:~/kes# 
root@ubuntu:~/kes# 
root@ubuntu:~/kes# printenv | grep KES_SERVER
KES_SERVER=https://127.0.0.1:7373
root@ubuntu:~/kes#

MinIO Server Setup:

  • Create kes-minio secret:

    • Copy ~/kes/minio.key and ~/kes/minio.crt to your laptop from the ubuntu pod

    • These two files below comes from this line: kes identity new --key minio.key --cert minio.crt minio:

# /Users/cniackz/minio/private.key is ~/kes/minio.key
# /Users/cniackz/minio/public.crt is ~/kes/minio.crt
kubectl create secret generic kes-minio -n tenant-lite --from-file=/Users/cniackz/minio/private.key --from-file=/Users/cniackz/minio/public.crt

  • Create kes-minio-public secret:

    • Copy ~/kes/private.key and ~/kes/public.crt to your laptop from the ubuntu pod

    • These two files below comes from this line: kes identity new --ip "10.244.2.7" localhost:

# /Users/cniackz/minio/private.key is ~/kes/private.key
# /Users/cniackz/minio/public.crt is ~/kes/public.crt
kubectl create secret generic kes-minio-public -n tenant-lite --from-file=/Users/cniackz/minio/private.key --from-file=/Users/cniackz/minio/public.crt

k edit tenant -n tenant-lite

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: storage
  namespace: minio-tenant
spec:
  # externalClientCertSecrets is to share the secret with the MinIO Pods:
  # Under: /tmp/certs/client-0 You will find:
  # client.crt and client.key
  # And we can use these files to setup KES in k8s
  externalClientCertSecrets:
  - name: kes-minio
    type: Opaque
  - name: kes-minio-public
    type: Opaque
  env:
    # Set MINIO_KMS_KES_ENDPOINT
    # It is the IP of the Ubuntu Pod.
    - name: MINIO_KMS_KES_ENDPOINT
      value: "https://<IP-ADDRESS-OF-UBUNTU-POD>:7373"
    # Set MinIO Client Credentials, it comes from kes-minio secret
    - name: MINIO_KMS_KES_CERT_FILE
      value: "/tmp/certs/client-0/client.crt"
    # Set MinIO Client Credentials, it comes from kes-minio secret
    - name: MINIO_KMS_KES_KEY_FILE
      value: "/tmp/certs/client-0/client.key"
    # Set MinIO Default Key
    - name: MINIO_KMS_KES_KEY_NAME
      value: "minio-default-key"
    # Trust the KES Server Certificate, it comes from kes-minio-public secret
    - name: MINIO_KMS_KES_CAPATH
      value: "/tmp/certs/client-1/client.crt"
    # Root User
    - name: MINIO_ROOT_USER
      value: minio
    # ROOT Password:
    - name: MINIO_ROOT_PASSWORD
      value: minio123

mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc mb myminio/my-bucket # create new bucket
mc admin kms key create myminio minio-my-bucket # create key
mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket

Result:

root@ubuntu:/# mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123
Added `myminio` successfully.
root@ubuntu:/# mc rb myminio/my-bucket --force # remove previous bucket to start fresh
mc: <ERROR> Unable to validate target `myminio/my-bucket`. Bucket `my-bucket` does not exist.
root@ubuntu:/# mc mb myminio/my-bucket # create new bucket
Bucket created successfully `myminio/my-bucket`.
root@ubuntu:/# mc admin kms key create myminio minio-my-bucket # create key
Created master key `minio-my-bucket` successfully
root@ubuntu:/# mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket
Auto encryption configuration has been set successfully for myminio/my-bucket
root@ubuntu:/#

Additional info:

  • Question:
KES is saving the keys in local memory right?, we are not using KMS at all, correct? Since the idea is to avoid KMS/Vault.
  • Answer:
KES is saving keys, identities and policies on a local drive - like MinIO stores objects in local  (and remote) drives.
Therefore, it is stateful. All state and persistence is maintained by KES itself. In case of stateless KES, KES relies on an external system - e.g. Vault - to
persist keys.
  • Question:
What are the advantages or disadvantages of using stateful kes in k8s? or in general...
  • Answer:
One major advantage of stateful KES is: It simplifies the deployment setup and reduces external dependencies.
In case of stateless KES, there must be some external KMS system. For example, you need to setup Vault first before you can deploy KES.
As a user/customer of MinIO you have to:
 1. Setup MinIO (and pay for MinIO's SubNet)
 2. Talk to an external KMS vendor and purchase their product as well.
You have two entities that have to work together and you probably pay both.
By using stateful KES, you can rule out the external KMS vendor since you get the KMS for free.
Whether stateful KES is beneficial for a customer depends upon whether they already have a (centrally managed) KMS solution.
If they do, then it may be easier to simply connect (stateless) KES to their KMS. Maybe their KMS powers other applications as well.
However, if they don't have such a solution, yet, or if it is not usable / isolated and they would have to purchase a KMS solution
in any case, then stateful KES can simplify their lives.
  • Question:
Why do we need to create KES unseal key? what is this for?
  • Answer:
As mentioned, stateful KES handles all data itself. Since it stores all crypto. keys, policies etc. locally it has to encrypt all data it writes to disk.
The unseal key is basically the key to unlock this encrypted storage.
Currently, KES supports a very simply unseal mechanism: There is a 256 bit encryption key which you provide via env. var.
However, KES is designed to support more complex unseal mechanisms in the future (there is a generic internal implementation).
For example, KES could implement unsealing via a hardware module - like TPM or HSM - or by talking to an external auth service.
The unsealing concept is not unique to KES. Any reasonable KMS implementation has to provide such a feature.
Hashicorp Vault defaults to an unseal mechanism called Secret Splitting. Instead of  one unseal key you have N (configurable)
unseal keys out of which you need M (configurable) unseal keys to unlock Vaults encrypted storage. It is a threshold mechansim
where e.g. 3 out of 5 unseal keys are required to unlock Vault.
KES could implement such a scheme too, but we decided not to for now.
When the unseal mechanism gets more complex then the initial learning curve for deploying gets significantly steeper. Therefore,
we went for the most basic unseal mechanism.
You specify the unseal mechanism in the kes init config file. Right now there is just environment but we can add more anytime in the future.

 unseal:
  environment:
    name: KES_UNSEAL_KEY
Clone this wiki locally