KES Encryption via UI

Introduction:

This wiki is just examples on how we test Server Side Encryption, lot of things can be improved and will be improved to make things much more clear for everyone. But in the meantime this is what we currently have and we really hope it helps people understand this a bit better when configuring! Happy reading 👍

Remember for Official docs, please go to links like: https://min.io/docs/minio/linux/operations/server-side-encryption/configure-minio-kes-aws.html where things are more organized but at the same time those are generic steps that requires more thinking to properly deploy something. But in the other hand, this wiki is not intended to replace offical docs, rather it is intended to complement it with some practical approach from the testing side.

Objective:

To show how to configure Server Side Encryption in MinIO for both types:

1. sse-s3 which is the easiest.
2. sse-kms which requires a bit more of work to configure.

Steps to configure sse-s3

Note: For sse-kms further steps need to be added, those extra steps are at the end of this wiki!

1. Delete cluster

kind delete clusters kind

2. Create new cluster:

kind create cluster --config ~/operator/testing/kind-config.yaml

3. Install Operator:

kubectl apply -k github.com/minio/operator/resources/\?ref\=v4.5.5

4. Deploy vault:

kubectl apply -f ~/operator/examples/vault/deployment.yaml

5. Wait for vault to be ready:

kubectl wait --namespace default \
	--for=condition=ready pod \
	--selector=app=vault \
	--timeout=120s

6. Get Vault Token:

VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault | grep "Root Token: " | sed -e "s/Root Token: //g")

You should see:

$ VAULT_ROOT_TOKEN=$(kubectl logs -l app=vault | grep "Root Token: " | sed -e "s/Root Token: //g")
$ echo $VAULT_ROOT_TOKEN
hvs.llM5QYQF9Cg10SowEudvf2j5

7. Run below command:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'

You should see:

$ kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault auth enable approle'
Success! Enabled approle auth method at: approle/

8. Run this command:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'

You should see:

$ kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault secrets enable kv'
Success! Enabled the kv secrets engine at: kv/

9. copy kes file

kubectl cp ~/operator/examples/vault/kes-policy.hcl $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}'):/kes-policy.hcl

10. Run:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'

You should see:

$ kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault policy write kes-policy /kes-policy.hcl'
Success! Uploaded policy: kes-policy

11. Run:

kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'

You should see:

$ kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy'
Success! Data written to: auth/approle/role/kes-role

12. Get Role ID:

ROLE_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault read auth/approle/role/kes-role/role-id' | grep "role_id    " | sed -e "s/role_id    //g")

You should see:

$ echo $ROLE_ID
003ea2c3-983d-e283-1f93-0c6baa2a710e

13. Get secret id:

SECRET_ID=$(kubectl exec $(kubectl get pods -l app=vault  | grep -v NAME | awk '{print $1}') -- sh -c 'VAULT_TOKEN='$VAULT_ROOT_TOKEN' VAULT_ADDR="http://127.0.0.1:8200" vault write -f auth/approle/role/kes-role/secret-id' | grep "secret_id             " | sed -e "s/secret_id             //g")

You should see:

$ echo $SECRET_ID
0f4289bd-6fb0-4271-483f-0c7d0d1cce44

14. Port forward Operator and access it:

kubectl -n minio-operator port-forward svc/console 9090

15. Apply the secret:

yq e 'select(.kind == "Secret")' "/Users/cniackz/operator/resources/base/console-ui.yaml" > tmp.yaml
kubectl apply -f tmp.yaml

16. Get the token

SA_TOKEN=$(kubectl -n minio-operator  get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
    echo "SA_TOKEN: ${SA_TOKEN}"

17. Get the cookie:

COOKIE=$(curl 'http://localhost:9090/api/v1/login/operator' -X POST -H 'Content-Type: application/json' --data-raw '{"jwt":"'$SA_TOKEN'"}' -i | grep "Set-Cookie: token=" | sed -e "s/Set-Cookie: token=//g" | awk -F ';' '{print $1}')

18. Create the tenant:

CREDENTIALS=$(curl 'http://localhost:9090/api/v1/tenants' -X POST -H 'Content-Type: application/json' -H 'Cookie: token='$COOKIE'' --data-raw '{"name":"kes-tenant","namespace":"default","access_key":"","secret_key":"","access_keys":[],"secret_keys":[],"enable_tls":true,"enable_console":true,"enable_prometheus":true,"service_name":"","image":"","expose_minio":true,"expose_console":true,"pools":[{"name":"pool-0","servers":4,"volumes_per_server":1,"volume_configuration":{"size":26843545600,"storage_class_name":"standard"},"securityContext":null,"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"v1.min.io/tenant","operator":"In","values":["kes-tenant"]},{"key":"v1.min.io/pool","operator":"In","values":["pool-0"]}]},"topologyKey":"kubernetes.io/hostname"}]}}}],"erasureCodingParity":2,"logSearchConfiguration":{"image":"minio/operator:dev","postgres_image":"","postgres_init_image":""},"prometheusConfiguration":{"image":"","sidecar_image":"","init_image":""},"tls":{"minio":[],"ca_certificates":[],"console_ca_certificates":[]},"encryption":{"replicas":"1","securityContext":{"runAsUser":"1000","runAsGroup":"1000","fsGroup":"1000","runAsNonRoot":true},"image":"","vault":{"endpoint":"http://vault.default.svc.cluster.local:8200","engine":"","namespace":"","prefix":"my-minio","approle":{"engine":"","id":"'$ROLE_ID'","secret":"'$SECRET_ID'","retry":0},"tls":{},"status":{"ping":0}}},"idp":{"keys":[{"access_key":"console","secret_key":"console123"}]}}')

Additional Info:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  creationTimestamp: "2022-11-29T00:19:09Z"
  generation: 1
  name: kes-tenant
  namespace: default
  resourceVersion: "3113"
  uid: f87ae7da-cd2a-4894-93dd-e1984e1fb57c
scheduler:
  name: ""
spec:
  configuration:
    name: kes-tenant-env-configuration
  credsSecret:
    name: kes-tenant-secret
  exposeServices:
    console: true
    minio: true
  image: minio/minio:RELEASE.2022-11-26T22-43-32Z
  imagePullSecret: {}
  kes:
    image: minio/kes:v0.17.6
    kesSecret:
      name: kes-tenant-secret-kes-configuration
    replicas: 1
    resources: {}
    securityContext:
      fsGroup: 1000
      fsGroupChangePolicy: Always
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
  log:
    audit:
      diskCapacityGB: 5
    db:
      resources: {}
      volumeClaimTemplate:
        metadata:
          name: kes-tenant-log
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: "5368709120"
        status: {}
    image: minio/operator:dev
    resources: {}
  mountPath: /export
  pools:
  - affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: v1.min.io/tenant
              operator: In
              values:
              - kes-tenant
            - key: v1.min.io/pool
              operator: In
              values:
              - pool-0
          topologyKey: kubernetes.io/hostname
    name: pool-0
    resources: {}
    servers: 4
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "26843545600"
        storageClassName: standard
      status: {}
    volumesPerServer: 1
  prometheus:
    diskCapacityGB: 5
    resources: {}
  requestAutoCert: true
  users:
  - name: kes-tenant-user-0
status:
  availableReplicas: 0
  certificates:
    autoCertEnabled: true
  currentState: Waiting for Log Search Pods to be ready
  drivesOnline: 4
  healthStatus: green
  pools:
  - legacySecurityContext: false
    ssName: kes-tenant-pool-0
    state: PoolInitialized
  revision: 0
  syncVersion: v4.5
  usage:
    capacity: 118908764160
    rawCapacity: 107374182400
    rawUsage: 73696313344
  writeQuorum: 3

Using UI rather than curl command:

If you want to use the UI rather than curl command to create the tenant, then follow this screens:

• When creating the tenant make sure to enable Server-Side Encryption also known as SSE:

• And put these values below:

KMS: Vault
Endpoint: http://vault.default.svc.cluster.local:8200
Prefix: my-minio

AppRole ID: It comes from vault configuration, look at lines below: 

====================================
$ echo $ROLE_ID
e6023d6f-f719-8b0a-9b16-291ec872dd95
====================================

AppRole Secret: Similar to above, it comes from vault configuration, look at lines below:

====================================
$ echo $SECRET_ID
c5d9c96a-ed1f-a9c2-9605-ee5328df4449
====================================

Replicas: 1

Run As User* 1000
Run As Group* 1000
FsGroup* 1000
Do not run as root is true

Things to expect:

1. Tenant should get green color and initialized state:

2. These pods are expected in the tenant namespace, here I used default as the namespace:

3. Kes pod, should look like below and it should be able to reach vault pod at http://vault.default.svc.cluster.local:8200, obviously that address and port will depend on your own vault pod or external vault from Hashicorp.:

Authenticating to Hashicorp Vault 'http://vault.default.svc.cluster.local:8200' ... 
Endpoint: https://127.0.0.1:7373        https://10.244.3.2:7373       

Admin:    _     [ disabled ]
Auth:     off   [ any client can connect but policies still apply ]

Keys:     Hashicorp Vault: http://vault.default.svc.cluster.local:8200

CLI:      export KES_SERVER=https://127.0.0.1:7373
          export KES_CLIENT_KEY=<client-private-key>   // e.g. $HOME/root.key
          export KES_CLIENT_CERT=<client-certificate>  // e.g. $HOME/root.cert
          kes --help

4. Vault logs normally looks like and notice the warning from the example, so please read more from Vault and configure accordingly:

2023-02-04T15:27:25.778Z [INFO]  secrets.kv.kv_cee97efd: upgrading keys finished
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variables:

    $ export VAULT_ADDR='http://0.0.0.0:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: uK8cxoVNYYcPtJim5Et18O1gAiEWJbZbwaaSVyFG3nQ=

5. MinIO Logs should look like and bucket are accesible via mc clinet or Console UI:

Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 119
All MinIO sub-systems initialized successfully in 1.553276126s
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-01-31T02-24-19Z (go1.19.4 linux/arm64)

Status:         16 Online, 0 Offline. 
API: https://minio.default.svc.cluster.local 
Console: https://10.244.3.7:9443 https://127.0.0.1:9443  

Test Server Side Encryption is actually working:

• This is for SSE-KMS also known as Server Side Encryption KMS:

root@ubuntu:/# mc encrypt set SSE-KMS "my-minio-key" myminio/encryption-bucket
Auto encryption configuration has been set successfully for myminio/encryption-bucket
root@ubuntu:/# mc encrypt info myminio/encryption-bucket
Auto encrytion 'sse-kms' is enabled with KeyID: my-minio-key
root@ubuntu:/# 
root@ubuntu:/# 
root@ubuntu:/# touch a.txt
root@ubuntu:/# mc cp a.txt myminio/encryption-bucket
 0 B / ? ━┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉┉━━root@ubuntu:/# 
root@ubuntu:/# 
root@ubuntu:/# 
root@ubuntu:/# mc stat myminio/encryption-bucket/a.txt
Name      : a.txt
Date      : 2023-02-04 16:17:32 UTC 
Size      : 0 B    
ETag      : aa455b31ab0ca823539c1d6e6cacdf20 
Type      : file 
Metadata  :
  Content-Type: text/plain 
Encrypted :
  X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:my-minio-key 
  X-Amz-Server-Side-Encryption               : aws:kms 

root@ubuntu:/# 

To configure SSE-KMS in k8s:

• Add kmskey as shown below in Encryption Raw Edition at our UI, this will simply update the secret of the configuration, remember if secret is already created delete the secret to be recreated by us from the UI:

• Secret looks like this:

• Then, also provided the variable to MinIO in the Tenant Spec:

• MinIO Pods get restarted and env var added, when all these is done, then SSE-KMS can be configured from mc: