Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add csi-s3 storage class as big data passing storage #753

Merged
merged 4 commits into from
Nov 2, 2021

Conversation

Tomcli
Copy link
Member

@Tomcli Tomcli commented Oct 14, 2021

Which issue is resolved by this Pull Request:
Resolves #569

Description of your changes:

  • Add CSI-S3 storage class third party plugins. (Translated from Datashim's CSI S3 helm chart to kustomize)
  • Add storage class configuration to big data passing (default to s3 storage), this way it doesn't need to provision new cloud storage for every new PVC.
  • Use in conjunction with Add artifact mid path to big data passing #751 to store the artifact in a centralized S3 artifact storage such as minio and COS.

Environment tested:

  • Python Version (use python --version):
  • Tekton Version (use tkn version):
  • Kubernetes Version (use kubectl version):
  • OS (e.g. from /etc/os-release):

Checklist:

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Tomcli
Copy link
Member Author

Tomcli commented Oct 14, 2021

@Udiknedormin This is the open source S3 CSI storage we use for mounting S3 storage. If there are other storages you want to use for mounting S3, please let us know. Thanks.

@Tomcli
Copy link
Member Author

Tomcli commented Oct 22, 2021

@ScrapCodes can you put lgtm here if this PR looks good to you?

@Tomcli
Copy link
Member Author

Tomcli commented Oct 23, 2021

/hold
the datashim team has some conflicts with this deployment. Need to rename the storageclass.

@ScrapCodes
Copy link
Contributor

Hi @Tomcli ,

Somehow, I am not succesfull in installing on IBM Cloud k8s cluster.

So, tried on a standalone instance.

After succesfully applying the following commands,

kubectl apply -k cluster-scoped-resources/
kubectl apply -k "env/platform-agnostic/" 
kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s

And trying to run the big_data_passing pipeline locally, I am getting POds in pending

          Message:               pod status "PodScheduled":"False"; message: "0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims."

May be CSI-s3 does not work on a standalone instance.

(.venv) 1 prashantsharma@prashant:~/go/pkg/src/github.com/kubeflow/pipelines$ kubectl logs -l app=csi-provisioner-s3  -n kubeflow
W1025 15:48:57.371227       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:07.370712       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:17.370715       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:27.370715       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:37.370727       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:47.370692       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:49:57.370711       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:50:07.370688       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:50:17.371190       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W1025 15:50:27.370714       1 connection.go:172] Still connecting to unix:///csi/csi.sock

@Tomcli
Copy link
Member Author

Tomcli commented Oct 29, 2021

/unhold

@Tomcli
Copy link
Member Author

Tomcli commented Oct 29, 2021

kubectl apply -k "env/platform-agnostic/"

Sorry with the late reply. kubectl apply -k "env/platform-agnostic/" should work if you are using a kubectl client that is v1.20+. I updated that as part of the developer readme.

Copy link
Contributor

@ScrapCodes ScrapCodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me !

Here we changed the default storage class to kfp-csi-s3, but the manifests installs a storage class csi-s3. Do you think we can add this point to documentation ?
Or can we change the manifests to install a storage class kfp-csi-s3 by default.

Looks like a I missed a change, LGTM!!

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ScrapCodes, Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ScrapCodes
Copy link
Contributor

/hold

@ScrapCodes
Copy link
Contributor

/lgtm

Feel free to unhold, if you think this is good to go.

@ScrapCodes
Copy link
Contributor

/unhold

@google-oss-robot google-oss-robot merged commit 424c79e into kubeflow:master Nov 2, 2021
@ScrapCodes
Copy link
Contributor

@Tomcli
Is it normal to see a very high capacity for the PVC

k get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-0fc0312d1f   Bound    pvc-9b269ff0-9efb-41eb-a714-812be702d950   9314Gi     RWO            kfp-csi-s3     18s

Even though the configured capacity is:

env | grep -i default
DEFAULT_ACCESSMODES=ReadWriteOnce
DEFAULT_STORAGE_CLASS=kfp-csi-s3
DEFAULT_STORAGE_SIZE=2Gi

@Tomcli
Copy link
Member Author

Tomcli commented Nov 2, 2021

@Tomcli Is it normal to see a very high capacity for the PVC

k get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-0fc0312d1f   Bound    pvc-9b269ff0-9efb-41eb-a714-812be702d950   9314Gi     RWO            kfp-csi-s3     18s

Even though the configured capacity is:

env | grep -i default
DEFAULT_ACCESSMODES=ReadWriteOnce
DEFAULT_STORAGE_CLASS=kfp-csi-s3
DEFAULT_STORAGE_SIZE=2Gi

I think that's the default behavior of the S3 CSI because the S3 storage can be expanded without updating the PVC. (E.g. when using IBM Cloud object storage). So they used the maximum capacity for pvc as default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support small K8s volume using non cloud service based storageclass
3 participants