RBD Helm Chart upgrade breaks with v3.3.x & attacher.enabled: false

[plnordquist]

Describe the bug

When upgrading to v3.3.0, Helm fails to patch the CSIDriver custom resource since attachRequired is an immutable field. I was able to work around this issue by deleting the CSIDriver custom resource and letting Helm re-create the resource when it upgraded the chart. I was also reviewing the chart and there are still pieces of the chart that reference the provisioner.attacher.enabled flag which probably don't make sense anymore since the CSIDriver resource is always requiring the attacher.

I'm also not sure if changing the attachRequired flag affects anything with already attached/mounted PVC objects. I didn't see any odd behavior by enabling the attacher when it was disabled but I didn't test it thoroughly.

Environment details

• Image/version of Ceph CSI driver : v3.2.1 -> 3.3.1
• Helm chart version : v3.2.1 -> 3.3.1
• Kernel version : 3.10.0-1160.21.1.el7.x86_64
• Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its
  krbd or rbd-nbd) : krbd
• Kubernetes cluster version : v1.19.10
• Ceph cluster version : 14.2.11

Steps to reproduce

Steps to reproduce the behavior:

1. Install the Helm chart version 3.2.1 with provisioner.attacher.enabled set to false
2. Attempt to upgrade the Helm chart to 3.3.1
3. See helm upgrade error

Additional context

The changeset that changes the behavior is here fba6a2d0.

[yati1998]

@plnordquist Can you please share the logs having the upgrade error?

[plnordquist]

Upgrade logs:

# helm upgrade -n ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --version 3.3.1 --values values.yaml
Error: UPGRADE FAILED: cannot patch "rbd.csi.ceph.com" with kind CSIDriver: CSIDriver.storage.k8s.io "rbd.csi.ceph.com" is invalid: spec: Invalid value: storage.CSIDriverSpec{AttachRequired:(*bool)(0xc0293f9980), FSGroupPolicy:(*storage.FSGroupPolicy)(nil), PodInfoOnMount:(*bool)(0xc0293f9981), VolumeLifecycleModes:[]storage.VolumeLifecycleMode{"Persistent"}, StorageCapacity:(*bool)(nil)}: field is immutable

Values:

csiConfig:
- clusterID: <clusterid>
  monitors:
  - <mon1>
  - <mon2>
  - <mon3>
nodeplugin:
  podSecurityPolicy:
    enabled: true
provisioner:
  attacher:
    enabled: false
  podSecurityPolicy:
    enabled: true
  tolerations:
  - effect: NoSchedule
    operator: Exists

Obviously continuing to keep provisioner.attacher.enabled set to false would not make sense so I would remove that value but it has to be there in 3.2.1 to trigger the issue.

[Madhu-1]

@plnordquist Yes, we should have documented this one, Currently we don't have any documentation of helm options and helm upgrade, I have opened #1076. Not enabling attacher causes issues related to RWO volume we need to enable attacher by default.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.