run kubernetes csi-test suite on ceph-csi

[Madhu-1]

we need to integrate ceph-csi with csi-test to test the CSI API capability of a driver.

[Madhu-1]

@nixpanic

[Madhu-1]

CC @ShyamsundarR

[ShyamsundarR]

csi-test plugin sanity runs by acting like the kubernetes driver services (provisioner/attacher/snapshotter etc.). IOW it sends in the required RPCs that invoke CSI plugin functionality across supported calls.

Hence, it does not need a kubernetes setup to run, or the kubernetes sidecars to function, it needs a running plugin instance to send the RPCs to. Also, it does not, as a result, pollute the CO system it is running on (in case it is used in such a manner).

Hence to run this against ceph-csi, without mocking any interface in ceph-csi plugins, we need the following prerequisites satisfied,
Prerequisites:

• A working ceph cluster, to point the plugin to perform its operations
• A environment with required ceph CLIs such that the plugin can make its CLI calls when running in such an environment
  • This can be a standalone node, where we bring up the plugin binary, OR
  • A kubenetes environment, where we run the plugin image as a pod (no csi sidecars required)

To run it in the above environment,

• Start the plugin (rbd or cephfs) with the required CLI arguments
  • Make sure to note the socket path that the plugin will listen on, as that is to be used as a parameter to the sanity test utility
  • If running standalone (outside kubernetes), all options (like, metadataStorage) that refer to kubernetes objects, should be configured to point to node specific locations
• Run the csi-sanity command like so, csi-sanity -ginkgo.v --csi.endpoint=/<socket-path>/csi-test.sock --csi.testvolumeparameters=./sc.yaml where sc.yaml contains all the required "YAML file of volume parameters for provisioned volumes"
  • In my case for stateless testing this yaml contained the following:

---
clusterID: rookcluster
pool: replicapool
imageFormat: "2"
imageFeatures: layering

• It is also required to have a secrets file, which I did not use, as I was testing the stateless implementation which uses secrets from the cluster-id secrets, and does not need it separately

NOTE: Current pass rate is about 50% with some low hangers that we can fix to improve the same.

---

Thoughts on integration:

• Run this for each PR
• Environment to run this in needs some thought, as we need the prerequisites sorted out
• Extend this as a framework, or use a similar framework to test out plugin internals by doing the required mocking or operating on the ceph cluster outside the plugin and forcing certain edge conditions to ensure plugin sanity closer to a unit test level test suite

Other frameworks that test CSI plugin functionality from a kubernetes standpoint also help, but to test out ceph-csi specific internals, IMO we need to start here and dig down into the code for other edge conditions, than use broader integration test frameworks to achieve the same.

[ShyamsundarR]

Here are the pod and RBAC manifests used against v1.0.0 rbdplugin for a test.

rbdplugin-pod.yaml
rbdplugin-rbac.yaml
Output from the test

[ShyamsundarR]

As a part of broader CI possibility, played with running minikube on TravisCI and here is an initial run, that setups up rook on a minikube instance in TravisCI. The files and code are here, and we can possibly start from here and see how to run csi-test suite next on the same.

[leseb]

CircleCI config to run minikube https://github.com/ceph/cn/blob/master/.circleci/config.yml

[mykaul]

@ShyamsundarR - any updates?

[ShyamsundarR]

@ShyamsundarR - any updates?

I am not on this task. I provided information based on runs and setup that I did to test the same out, to aid adoption into the e2e framework.

I agree this is a priority, as it helps keep the CSI plugins code quality higher and should be tracked as such.

[Madhu-1]

@humblec @ShyamsundarR @poornimag @nixpanic anyone like to take this up for next release?

[nixpanic]

Not sure for what release this will be ready, but I'm working on this a little now.

Environment for local testing:

• vagrant VM with CentOS-7
• minikube inside the VM (with --vm-driver-none)
• Ceph Nano for minimal Ceph environment
• Ceph-CSI from local git repository with last commit

This should allow for easy integration in the CentOS CI or other CI environments.

[Madhu-1]

@nixpanic how about running this as a kubernetes Job in E2E testing?

we should be able to download the csi-sanity binary and connect to the socket where csi driver is running

[nixpanic]

More details are in https://kubernetes.io/blog/2020/01/08/testing-of-csi-drivers/

[humblec]

@nixpanic @Madhu-1 do we need this any more ?

[Madhu-1]

Yes we need to run csi-test with cephcsi

[humblec]

Yes we need to run csi-test with cephcsi

as long as we cover the features with e2e i miss the point why we need that :)

[Madhu-1]

Yes we need to run csi-test with cephcsi

as long as we cover the features with e2e i miss the point why we need that :)

to make sure we return proper error code and to make sure we are not missing any other valid or invalid cases

[Madhu-1]

@nixpanic do we need it in release-3.0.0 milestone? or can we move it outside

[Madhu-1]

@nixpanic can this be done in 3.1.0? or do we need to move it out to the next release?

[humblec]

Moving this from release 3.1.0

[humblec]

@yati1998 is this something you would like to explore ?

[yati1998]

@humblec sure I will take a look in this.

[yati1998]

@yati1998 is this something you would like to explore ?

@humblec it will be difficult for me to work on this as of now. If it is a priority issue, it will be better to assign it to someone who have experience on this. Sorry for mentioning this so late.

[nixpanic]

I'm moving this out of 3.4 in favour of #2015

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[humblec]

/assign @humblec