How to install directpv in added nodes

Objective:

To show/provide a demo on how you can install DirectPV in new added nodes to an existing cluster.

Steps:

Please read all documentation on how to upgrade DirectPV if not in the latest version:

Once the documentation has been read, please proceed and upgrade accordingly.

For this demo, initial version was:

minio@minio-k8s17:~$ kubectl directpv --version
directpv version v3.2.0

We uninstall it:

kubectl directpv uninstall

Expected:

minio@minio-k8s17:~$ kubectl directpv uninstall
I0413 15:07:00.092622 2844108 deployment.go:302] 'direct.csi.min.io' controller deployment deleted
I0413 15:07:00.098556 2844108 daemonset.go:53] 'direct.csi.min.io' daemonset deleted
I0413 15:07:00.110232 2844108 service.go:47] 'direct.csi.min.io' service deleted
I0413 15:07:00.113472 2844108 validationrules.go:155] 'direct.csi.min.io' validation rules removed
I0413 15:07:01.184327 2844108 storageclass.go:60] 'direct.csi.min.io' storageclass deleted
I0413 15:07:01.967597 2844108 csidriver.go:158] 'direct.csi.min.io' csidriver deleted
I0413 15:07:01.974746 2844108 conversion_secret.go:42] 'direct.csi.min.io' conversion secrets deleted
W0413 15:07:01.987550 2844108 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I0413 15:07:01.987604 2844108 psp.go:163] 'direct.csi.min.io' pod security policy removed
I0413 15:07:02.052471 2844108 rbac.go:377] 'direct.csi.min.io' rbac roles deleted
I0413 15:07:02.066227 2844108 ns.go:82] 'direct.csi.min.io' namespace deleted

We make sure no remaining pod is still present in the cluster.
Uninstalled the plugin:

minio@minio-k8s17:~$ kubectl krew uninstall directpv
Uninstalled plugin: directpv

Then re-installed the plugin:

minio@minio-k8s17:~$ kubectl krew install directpv
Updated the local copy of plugin index.
Installing plugin: directpv
Installed plugin: directpv
\
 | Use this plugin:
 | 	kubectl directpv
 | Documentation:
 | 	https://github.com/minio/directpv
/
WARNING: You installed plugin "directpv" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

Labeled the nodes to show how you can select the nodes where DirectPV will run:

kubectl label nodes minio-k8s17 directpv=yes
kubectl label nodes minio-k8s18 directpv=yes
kubectl label nodes minio-k8s19 directpv=yes
kubectl label nodes minio-k8s20 directpv=yes

Finally the installation is made against those nodes:

minio@minio-k8s17:~$ kubectl directpv install --node-selector directpv=yes

 ███████████████████████████████████████████████████████████████████████████ 100%

┌──────────────────────────────────────┬──────────────────────────┐
│ NAME                                 │ KIND                     │
├──────────────────────────────────────┼──────────────────────────┤
│ directpv                             │ Namespace                │
│ directpv-min-io                      │ ServiceAccount           │
│ directpv-min-io                      │ ClusterRole              │
│ directpv-min-io                      │ ClusterRoleBinding       │
│ directpv-min-io                      │ Role                     │
│ directpv-min-io                      │ RoleBinding              │
│ directpv-min-io                      │ PodSecurityPolicy        │
│ psp-directpv-min-io                  │ ClusterRoleBinding       │
│ directpvdrives.directpv.min.io       │ CustomResourceDefinition │
│ directpvvolumes.directpv.min.io      │ CustomResourceDefinition │
│ directpvnodes.directpv.min.io        │ CustomResourceDefinition │
│ directpvinitrequests.directpv.min.io │ CustomResourceDefinition │
│ directpv-min-io                      │ CSIDriver                │
│ direct-csi-min-io                    │ CSIDriver                │
│ directpv-min-io                      │ StorageClass             │
│ direct-csi-min-io                    │ StorageClass             │
│ node-server                          │ Daemonset                │
│ legacy-node-server                   │ Daemonset                │
│ controller                           │ Deployment               │
│ legacy-controller                    │ Deployment               │
└──────────────────────────────────────┴──────────────────────────┘

DirectPV installed successfully

Version is now:

minio@minio-k8s17:~$ kubectl directpv --version
directpv version v4.0.3

And DirectPV is running in selected nodes:

We can see all drives:

minio@minio-k8s17:~$ kubectl directpv list drives
┌─────────────┬──────────┬─────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE        │ NAME     │ MAKE                │ SIZE   │ FREE   │ VOLUMES │ STATUS │
├─────────────┼──────────┼─────────────────────┼────────┼────────┼─────────┼────────┤
│ minio-k8s17 │ nvme1n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s17 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s17 │ nvme2n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme3n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme4n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s17 │ nvme5n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s17 │ nvme6n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme7n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s17 │ nvme8n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s17 │ nvme9n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme3n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4       │ Ready  │
│ minio-k8s18 │ nvme8n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 11 TiB │ 4       │ Ready  │
│ minio-k8s18 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme1n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme2n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme4n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme5n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme6n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme7n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s18 │ nvme9n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s19 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s19 │ nvme1n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme2n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme3n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s19 │ nvme4n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s19 │ nvme5n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme6n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 5       │ Ready  │
│ minio-k8s19 │ nvme7n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s19 │ nvme8n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 5       │ Ready  │
│ minio-k8s19 │ nvme9n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme10n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme11n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s20 │ nvme12n1 │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme1n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme2n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 2       │ Ready  │
│ minio-k8s20 │ nvme3n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme4n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme5n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s20 │ nvme6n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 4       │ Ready  │
│ minio-k8s20 │ nvme7n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 2       │ Ready  │
│ minio-k8s20 │ nvme8n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
│ minio-k8s20 │ nvme9n1  │ INTEL SSDPE2NV153T8 │ 14 TiB │ 12 TiB │ 3       │ Ready  │
└─────────────┴──────────┴─────────────────────┴────────┴────────┴─────────┴────────┘

And since this is existing DirectPV cluster, no initialization is longer needed:

minio@minio-k8s17:~$ kubectl directpv discover

 Discovered node 'minio-k8s17' ✔
 Discovered node 'minio-k8s18' ✔
 Discovered node 'minio-k8s19' ✔
 Discovered node 'minio-k8s20' ✔

No drives are available to initialize

Test DirectPV can provision Persistent Volumes:

Use this claim as example and modify it accordingly, call it claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: please-remove-me-cesar
  namespace: ns-3
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: direct-csi-min-io
    volume.kubernetes.io/selected-node: minio-k8s18
  finalizers:
    - kubernetes.io/pvc-protection
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: directpv-min-io
  volumeMode: Filesystem

Apply it:

kubectl apply -f claim.yaml

Check it gets Bound:

$ k get pv pvc-49ad809a-9521-41a1-a9f4-20fb0b007194
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS      REASON   AGE
pvc-49ad809a-9521-41a1-a9f4-20fb0b007194   5Gi        RWO            Delete           Bound    ns-3/please-remove-me-cesar   directpv-min-io            94s
Cesars-MacBook-Pro:~ cniackz$

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to install directpv in added nodes

Objective:

Steps:

Clone this wiki locally