helm: support encryption config in ceph-csi-cephfs chart #4531

acolombier · 2024-04-01T23:11:09Z

This PR adds encryption configuration to the cephfs-csi chart, similarly to the RBD one,

This fixes permission missing when using the kubernetes KMS

Related issues

Fixes: ##4470

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
[?] Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
[-] Unit tests have been added, if necessary.
[-] Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

acolombier · 2024-04-01T23:11:38Z

Let me know if you would need me to update the Pending release notes for this PR

nixpanic

minor changes needed to pass CI/yamllint

charts/ceph-csi-cephfs/values.yaml

nixpanic · 2024-04-16T15:46:44Z

yamllint passed, now MarkDown linting needs to be fixed as well...

acolombier · 2024-04-17T08:49:38Z

Apologies for missing this previously @nixpanic . Is it expected that the pre-commit check doesn't run any of the checkers? Or is my local setup flawed?

nixpanic · 2024-04-17T15:42:57Z

Apologies for missing this previously @nixpanic . Is it expected that the pre-commit check doesn't run any of the checkers? Or is my local setup flawed?

I don't think pre-commit runs all the checks. I usually run make containerized-test to check if everything is OK.

Madhu-1 · 2024-04-18T12:49:38Z

charts/ceph-csi-cephfs/templates/nodeplugin-clusterrole.yaml

@@ -3,6 +3,7 @@ kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
  name: {{ include "ceph-csi-cephfs.nodeplugin.fullname" . }}
+  namespace: {{ .Release.Namespace }}


we dont need namespace for clusterRole

Madhu-1 · 2024-04-18T12:52:13Z

charts/ceph-csi-cephfs/templates/nodeplugin-role.yaml

@@ -0,0 +1,21 @@
+{{- if .Values.rbac.create -}}
+{{- if and .Values.encryptionKMSConfig .Values.encryptionKMSConfig.secretNamespace  .Values.encryptionKMSConfig.secretName (eq .Values.encryptionKMSConfig.secretNamespace .Release.Namespace) -}}
+kind: Role


we dont need Role and Role Binding right? we need to have access to secrets in different namespaces as well. i don't see Role https://github.com/ceph/ceph-csi/tree/devel/deploy/cephfs/kubernetes, am i missing anything?

The idea was to downgrade the ClusterRole to just a Role in case the configured secret in encryptionKMSConfig was local to the namespace. This was done to help with least privileged and limit the access to secret as much as possible. Happy to remove if you think this is overkill. Arguably, users who want least privileged roles can create manage the RBAC themselves

Ceph-CSI components which reads the Secrets are usually running in a different namespace than the namespace where Secrets for applications have their encryption key. Ceph-CSI needs to read the Secret from a different namespace than where it is running/mounting the PV.

In this case, it will create the ClusterRole automatically. The logic at line 2 is if encryptionKMSConfig.secretNamespace == Release.Namespace, then create a Role instead of adding secret:read to the ClusterRole.

Happy to scratch that optimisation, and create RBAC myself! But just thought it would be a harmless optimisation to put in

I have slightly reworked the logic behind it to enforce determinism and added a some documentation in the REAMDE.md. I'd be nice if you could tell me how that looks, and if you like it, I can spin up a PR with the same feature for RBD.

@nixpanic it'd be great if you could let me know your though on this

@acolombier @nixpanic is on PTO will be back 1st week of Jun, do you want me to block this PR until he is back to confirm on this?

Up to you! If it is ready to be merged, happy to do so too, and I'll wait and see if @nixpanic likes this approach for RBD and I can make the PR then

@nixpanic friendly ping in case you missed the notification :)

Pull request has been modified.

Madhu-1

LGTM, Thanks

Pull request has been modified.

acolombier · 2024-05-15T12:01:42Z

Not sure what is going on with commitlint - is that something that needs to be fixed on my end? Do you want me to merge devel?

Edit my commit to fix mdlint - sorry @Madhu-1

Madhu-1 · 2024-05-15T12:06:01Z

@acolombier can you please rebase the PR on top of devel branch?

acolombier · 2024-05-15T12:06:57Z

Done @Madhu-1

Madhu-1 · 2024-05-15T12:10:01Z

@Mergifyio rebase

ceph-csi-bot · 2024-11-06T14:30:03Z

/test ci/centos/k8s-e2e-external-storage/1.29

ceph-csi-bot · 2024-11-06T14:30:03Z

/test ci/centos/mini-e2e/k8s-1.30

ceph-csi-bot · 2024-11-06T14:30:04Z

/test ci/centos/mini-e2e/k8s-1.31

ceph-csi-bot · 2024-11-06T14:30:04Z

/test ci/centos/mini-e2e-helm/k8s-1.29

ceph-csi-bot · 2024-11-06T14:30:05Z

/test ci/centos/mini-e2e/k8s-1.29

acolombier · 2024-11-06T20:34:49Z

Thanks for not letting the PR die @nixpanic !

Let me know how/if I can fix those tests!

iPraveenParihar · 2024-11-07T10:16:15Z

/retest ci/centos/mini-e2e-helm/k8s-1.29

nixpanic · 2024-11-08T16:17:30Z

Let me know how/if I can fix those tests!

It seems to go wrong with the cephfs provisioner deployment, and maybe other parts.

https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.29/detail/mini-e2e-helm_k8s-1.29/514/pipeline/90

I suspect something sneaked in the helm charts that causes some problematic yaml generation.

Probably this:

Error: INSTALLATION FAILED: Unable to continue with install: could not get information about the resource ConfigMap "" in namespace "cephcsi-e2e-aa06ab4c126c": resource name may not be empty

Pull request has been modified.

acolombier · 2024-11-09T18:05:44Z

Thanks for the tip @nixpanic - indeed, as I tried to unify the way encryption works with the RBD chart, I forgot some chart values from there. Should be ready for testing now!

Will squash once the test passes if that's okay with you.

iPraveenParihar · 2024-11-11T04:38:43Z

/test ci/centos/mini-e2e-helm/k8s-1.29

iPraveenParihar · 2024-11-11T07:03:23Z

Error: INSTALLATION FAILED: Unable to continue with install: ConfigMap "ceph-csi-encryption-kms-config" in namespace "cephcsi-e2e-d1df94a5c755" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "ceph-csi-rbd": current value is "ceph-csi-cephfs"

@acolombier, the install-helm.sh script fails to install RBD driver. Could you please include the change to delete the configMap ceph-csi-encryption-kms-config in install-helm.sh?

ceph-csi/scripts/install-helm.sh

Lines 187 to 191 in cea8bf8

    
           # deleting configmaps as a workaround to avoid configmap already present 
        
           # issue when installing ceph-csi-rbd 
        
           kubectl_retry delete cm ceph-csi-config --namespace "${NAMESPACE}" 
        
           kubectl_retry delete cm ceph-config --namespace "${NAMESPACE}"

acolombier · 2024-11-11T09:14:01Z

Done @iPraveenParihar !

iPraveenParihar · 2024-11-11T09:15:00Z

/test ci/centos/mini-e2e-helm/k8s-1.29

iPraveenParihar · 2024-11-12T05:08:08Z

the helm test passed 🎉

iPraveenParihar · 2024-11-12T05:10:39Z

@acolombier, can you please squash your fixup! commits?

this chart currently lack the ability to properly configure encryption, as well as granting sufficent permission to allow controllers to access secret when needed. Signed-off-by: Antoine C <[email protected]>

this allows the encryption KMS config to be granted secret access with a least privilges policy. Signed-off-by: Antoine C <[email protected]>

acolombier · 2024-11-12T09:13:35Z

Done!

Pull request has been modified.

iPraveenParihar · 2024-11-14T11:33:54Z

@Mergifyio queue

mergify · 2024-11-14T11:34:07Z

queue

🛑 The pull request has been removed from the queue `default`

The pull request can't be updated.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

mergify bot added the component/deployment Helm chart, kubernetes templates and configuration Issues/PRs label Apr 1, 2024

acolombier force-pushed the feat/native-encryption-support branch 2 times, most recently from 80ba8b1 to 1dd0f21 Compare April 1, 2024 23:16

nixpanic reviewed Apr 9, 2024

View reviewed changes

charts/ceph-csi-cephfs/values.yaml Show resolved Hide resolved

charts/ceph-csi-cephfs/values.yaml Outdated Show resolved Hide resolved

acolombier force-pushed the feat/native-encryption-support branch from d8ff294 to 2f39774 Compare April 14, 2024 12:14

nixpanic requested a review from iPraveenParihar April 15, 2024 11:11

acolombier force-pushed the feat/native-encryption-support branch from 2f39774 to 923b160 Compare April 15, 2024 21:08

nixpanic added the enhancement New feature or request label Apr 16, 2024

nixpanic requested a review from a team April 16, 2024 07:26

acolombier force-pushed the feat/native-encryption-support branch from 923b160 to b8bba0d Compare April 17, 2024 08:48

nixpanic previously approved these changes Apr 18, 2024

View reviewed changes

nixpanic requested a review from a team April 18, 2024 07:05

Madhu-1 requested changes Apr 18, 2024

View reviewed changes

acolombier requested review from Madhu-1 and nixpanic May 15, 2024 11:27

Madhu-1 previously approved these changes May 15, 2024

View reviewed changes

Madhu-1 requested a review from Rakshith-R May 15, 2024 11:55

acolombier force-pushed the feat/native-encryption-support branch from 58ac9c4 to 7849b15 Compare May 15, 2024 12:00

acolombier requested a review from Madhu-1 May 15, 2024 12:01

acolombier force-pushed the feat/native-encryption-support branch from 7849b15 to 4fa3d47 Compare May 15, 2024 12:06

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Nov 6, 2024

iPraveenParihar previously approved these changes Nov 12, 2024

View reviewed changes

iPraveenParihar requested review from Madhu-1 and nixpanic November 12, 2024 05:09

acolombier added 2 commits November 12, 2024 09:13

helm: support encryption config in ceph-csi-cephfs chart

925f469

this chart currently lack the ability to properly configure encryption, as well as granting sufficent permission to allow controllers to access secret when needed. Signed-off-by: Antoine C <[email protected]>

helm: add least privileges logic for secrets on ceph-csi-cephfs chart

fa2c0f6

this allows the encryption KMS config to be granted secret access with a least privilges policy. Signed-off-by: Antoine C <[email protected]>

acolombier force-pushed the feat/native-encryption-support branch from ce39f32 to fa2c0f6 Compare November 12, 2024 09:13

Madhu-1 approved these changes Nov 12, 2024

View reviewed changes

iPraveenParihar approved these changes Nov 13, 2024

View reviewed changes

helm: support encryption config in ceph-csi-cephfs chart #4531

Are you sure you want to change the base?

helm: support encryption config in ceph-csi-cephfs chart #4531

Conversation

acolombier commented Apr 1, 2024

Related issues

acolombier commented Apr 1, 2024

nixpanic left a comment

Choose a reason for hiding this comment

nixpanic commented Apr 16, 2024

acolombier commented Apr 17, 2024 • edited Loading

nixpanic commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acolombier Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Madhu-1 left a comment

Choose a reason for hiding this comment

acolombier commented May 15, 2024 • edited Loading

Madhu-1 commented May 15, 2024

acolombier commented May 15, 2024

Madhu-1 commented May 15, 2024

ceph-csi-bot commented Nov 6, 2024

ceph-csi-bot commented Nov 6, 2024

ceph-csi-bot commented Nov 6, 2024

ceph-csi-bot commented Nov 6, 2024

ceph-csi-bot commented Nov 6, 2024

acolombier commented Nov 6, 2024 • edited Loading

iPraveenParihar commented Nov 7, 2024

nixpanic commented Nov 8, 2024 • edited Loading

acolombier commented Nov 9, 2024 • edited Loading

iPraveenParihar commented Nov 11, 2024

iPraveenParihar commented Nov 11, 2024

acolombier commented Nov 11, 2024

iPraveenParihar commented Nov 11, 2024

iPraveenParihar commented Nov 12, 2024

iPraveenParihar commented Nov 12, 2024

acolombier commented Nov 12, 2024

iPraveenParihar commented Nov 14, 2024

mergify bot commented Nov 14, 2024 • edited Loading

🛑 The pull request has been removed from the queue default

acolombier commented Apr 17, 2024 •

edited

Loading

acolombier Apr 18, 2024 •

edited

Loading

acolombier commented May 15, 2024 •

edited

Loading

acolombier commented Nov 6, 2024 •

edited

Loading

nixpanic commented Nov 8, 2024 •

edited

Loading

acolombier commented Nov 9, 2024 •

edited

Loading

mergify bot commented Nov 14, 2024 •

edited

Loading

🛑 The pull request has been removed from the queue `default`