Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[manila-csi-plugin] add support for share group and affinity annotations #2685

Open
kayrus opened this issue Oct 8, 2024 · 1 comment · May be fixed by #2687
Open

[manila-csi-plugin] add support for share group and affinity annotations #2685

kayrus opened this issue Oct 8, 2024 · 1 comment · May be fixed by #2687
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@kayrus
Copy link
Contributor

kayrus commented Oct 8, 2024

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

The Manila API allows to specify a share group in which the new share should be created. Along with this it supports affinity/anti-affinity scheduler hints.

What you expected to happen:

The share group ID can be specified in manila storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-manila-nfs
provisioner: nfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
  # Manila share type
  type: default
  # Manila share group
  group-id: %my-group-id%

The scheduler hints for a particular share can be specified in a PVC annotation like it's donw in AWS EBS CSI driver:

$ k describe pv           
Name:              pvc-6bff47f9-1843-4576-ba15-158c73491e8c
Labels:            <none>
Annotations:       ebs.csi.aws.com/iops: 4000
                   ebs.csi.aws.com/volumeType: io2

The proposed annotation keys are:

  • manila.csi.openstack.org/affinity: %shareID1%,%shareID2%
  • manila.csi.openstack.org/anti-affinity: %shareID3%,%shareID4%
  • manila.csi.openstack.org/group-id: %groupID% - will override the SC group ID

The PVC with the annotations below:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    manila.csi.openstack.org/affinity: a07a01c7-43b9-484a-8745-717e44d8f662,749ad593-0a99-4e32-91ba-7106db7ef792
    manila.csi.openstack.org/anti-affinity: 364f7c8c-7a60-4023-8458-b332e176789a,1101d1d5-b68d-4f17-8604-46271852b6db
    manila.csi.openstack.org/group-id: 5b5f4dce-6437-461e-80bb-77b0a1f3671f

and storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-manila-nfs
provisioner: nfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
  type: default
  group-id: ca8e84aa-a0c7-4631-9b82-69c95b50a4bd

will produce an API call below:

{
    "share": {
        "share_proto": "nfs",
        "share_network_id": "713df749-aac0-4a54-af52-10f6c991e80c",
        "share_group_id": "5b5f4dce-6437-461e-80bb-77b0a1f3671f",
        "name": "pvc_123",
        "size": 1,
        "scheduler_hints": {
            "same_host": "a07a01c7-43b9-484a-8745-717e44d8f662,749ad593-0a99-4e32-91ba-7106db7ef792",
            "different_host": "364f7c8c-7a60-4023-8458-b332e176789a,1101d1d5-b68d-4f17-8604-46271852b6db"
        }
    }
}

Accessing PVC annotations can be supported with a side-car modifier, see kubernetes-sigs/aws-ebs-csi-driver#1600 for example.

How to reproduce it:

n/a

Anything else we need to know?:

Upstream gophercloud feature request for the scheduler hints: gophercloud/gophercloud#3204

Additionally we can implement the VolumeAttributesClass support. VolumeAttributesClass parameters will override StorageClass parameters, and PVC annotations will override VolumeAttributesClass parameters. For the volumeAttributesClassName support manila CSI must have a k8s client configured. However this can be an extra task. See kubernetes-sigs/aws-ebs-csi-driver#1941 for reference. This approach won't require a side-car container to fetch PVC annotations.

Environment:

  • openstack-cloud-controller-manager(or other related binary) version:
  • OpenStack version:
  • Others:
@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 8, 2024
@kayrus
Copy link
Contributor Author

kayrus commented Oct 8, 2024

I reviewed the https://github.com/awslabs/volume-modifier-for-k8s sidecar code, and it seems there's no straightforward way to intercept the CreateVolume request without resorting to workarounds like mutexKV[volID]. When PVC annotations feature is enabled, the CreateVolume request must wait for the corresponding ModifyVolumeProperties request to be processed, which introduces complexity.

An alternative approach would be to embed a Kubernetes client inside the CSI controller and fetch annotations during the CreateVolume request.

Ideally, leveraging VolumeAttributesClass parameters would be a cleaner solution. However, due to affinity constraints (like the volume ID list), this parameter would likely be unique per PVC rather than a shared VolumeAttributesClass.

@kayrus kayrus linked a pull request Oct 14, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants