PVs not created on EFS #122
Comments
spiffxp
added a commit
to spiffxp/nfs-subdir-external-provisioner
that referenced
this issue
Sep 20, 2021
5489de6 Merge pull request kubernetes-sigs#174 from mauriciopoppe/bump-kind-version 0c675d4 Bump kind version to v0.11.1 ef69a88 Merge pull request kubernetes-sigs#173 from nick5616/add-ws2022 44c710c added WS2022 to build platforms 0883be4 Merge pull request kubernetes-sigs#171 from pohly/example-commands 02cda51 build.make: support binaries outside of cmd, with optional go.mod 65922ea Merge pull request kubernetes-sigs#170 from pohly/canary-snapshot-controller c0bdfb3 prow.sh: deploy canary snapshot-controller in canary jobs 0438f15 Merge pull request kubernetes-sigs#167 from c0va23/feature/release-armv7-image 4786f4d Merge pull request kubernetes-sigs#168 from msau42/update-release-prereq 6a2dc64 Remove requirement to be top-level approver. Only maintainers membership is required to do a release 30a4f7b Release armv7 image ac8108f Merge pull request kubernetes-sigs#165 from consideRatio/pr/update-github-links-ref-to-master-to-HEAD 999b483 docs: make github links reference HEAD instead of main fd67069 docs: make github links reference HEAD instead of master c0a4fb1 Merge pull request kubernetes-sigs#164 from anubha-v-ardhan/patch-1 9c6a6c0 Master to main cleanup 682c686 Merge pull request kubernetes-sigs#162 from pohly/pod-name-via-shell-command 36a29f5 Merge pull request kubernetes-sigs#163 from pohly/remove-bazel 68e43ca prow.sh: remove Bazel build support c5f59c5 prow.sh: allow shell commands in CSI_PROW_SANITY_POD 71c810a Merge pull request kubernetes-sigs#161 from pohly/mock-test-fixes 9e438f8 prow.sh: fix mock testing d7146c7 Merge pull request kubernetes-sigs#160 from pohly/kind-update 4b6aa60 prow.sh: update to KinD v0.11.0 7cdc76f Merge pull request kubernetes-sigs#159 from pohly/fix-deployment-selection ef8bd33 prow.sh: more flexible CSI_PROW_DEPLOYMENT, part II 204bc89 Merge pull request kubernetes-sigs#158 from pohly/fix-deployment-selection 61538bb prow.sh: more flexible CSI_PROW_DEPLOYMENT 2b0e6db Merge pull request kubernetes-sigs#157 from humblec/csi-release a2fcd6d Adding myself to csi reviewers group f325590 Merge pull request kubernetes-sigs#149 from pohly/cluster-logs 4b03b30 Merge pull request kubernetes-sigs#155 from pohly/owners a6453c8 owners: introduce aliases ad83def Merge pull request kubernetes-sigs#153 from pohly/fix-image-builds 5561780 build.make: fix image publishng 29bd39b Merge pull request kubernetes-sigs#152 from pohly/bump-csi-test bc42793 prow.sh: use csi-test v4.2.0 b546baa Merge pull request kubernetes-sigs#150 from mauriciopoppe/windows-multiarch-args bfbb6f3 add parameter base_image and addon_image to BUILD_PARAMETERS 2d61d3b Merge pull request kubernetes-sigs#151 from humblec/cm 48e71f0 Replace `which` command ( non standard) with `command -v` builtin feb20e2 prow.sh: collect cluster logs 7b96bea Merge pull request kubernetes-sigs#148 from dobsonj/add-checkpathcmd-to-prow 2d2e03b prow.sh: enable -csi.checkpathcmd option in csi-sanity 09d4151 Merge pull request kubernetes-sigs#147 from pohly/mock-testing 74cfbc9 prow.sh: support mock tests 4a3f110 prow.sh: remove obsolete test suppression 6616a6b Merge pull request kubernetes-sigs#146 from pohly/kubernetes-1.21 510fb0f prow.sh: support Kubernetes 1.21 c63c61b prow.sh: add CSI_PROW_DEPLOYMENT_SUFFIX 51ac11c Merge pull request kubernetes-sigs#144 from pohly/pull-jobs dd54c92 pull-test.sh: test importing csi-release-tools into other repo 7d2643a Merge pull request kubernetes-sigs#143 from pohly/path-setup 6880b0c prow.sh: avoid creating paths unless really running tests bc0504a Merge pull request kubernetes-sigs#140 from jsafrane/remove-unused-k8s-libs 5b1de1a go-get-kubernetes.sh: remove unused k8s libs 49b4269 Merge pull request kubernetes-sigs#120 from pohly/add-kubernetes-release a1e1127 Merge pull request kubernetes-sigs#139 from pohly/kind-for-kubernetes-latest 1c0fb09 prow.sh: use KinD main for latest Kubernetes 1d77cfc Merge pull request kubernetes-sigs#138 from pohly/kind-update-0.10 bff2fb7 prow.sh: KinD 0.10.0 95eac33 Merge pull request kubernetes-sigs#137 from pohly/fix-go-version-check 437e431 verify-go-version.sh: fix check after removal of travis.yml 1748b16 Merge pull request kubernetes-sigs#136 from pohly/go-1.16 ec844ea remove travis.yml, Go 1.16 df76aba Merge pull request kubernetes-sigs#134 from andyzhangx/add-build-arg e314a56 add build-arg ARCH for building multi-arch images, e.g. ARG ARCH FROM k8s.gcr.io/build-image/debian-base-${ARCH}:v2.1.3 7bc70e5 Merge pull request kubernetes-sigs#129 from pohly/squash-documentation e0b02e7 README.md: document usage of --squash 316cb95 Merge pull request kubernetes-sigs#132 from yiyang5055/bugfix/boilerplate 26e2ab1 fix: default boilerplate path 1add8c1 Merge pull request kubernetes-sigs#133 from pohly/kubernetes-1.20-tag 3e811d6 prow.sh: fix "on-master" prow jobs 1d60e77 Merge pull request kubernetes-sigs#131 from pohly/kubernetes-1.20-tag 9f10459 prow.sh: support building Kubernetes for a specific version f7e7ee4 docs: steps for adding testing against new Kubernetes release fe1f284 Merge pull request kubernetes-sigs#121 from kvaps/namespace-check 8fdf0f7 Merge pull request kubernetes-sigs#128 from fengzixu/master 1c94220 fix: fix a bug of csi-sanity a4c41e6 Merge pull request kubernetes-sigs#127 from pohly/fix-boilerplate ece0f50 check namespace for snapshot-controller dbd8967 verify-boilerplate.sh: fix path to script 9289fd1 Merge pull request kubernetes-sigs#125 from sachinkumarsingh092/optional-spelling-boilerplate-checks ad29307 Make the spelling and boilerplate checks optional 5f06d02 Merge pull request kubernetes-sigs#124 from sachinkumarsingh092/fix-spellcheck-boilerplate-tests 48186eb Fix spelling and boilerplate errors 71690af Merge pull request kubernetes-sigs#122 from sachinkumarsingh092/include-spellcheck-boilerplate-tests 981be3f Adding spelling and boilerplate checks. 2bb7525 Merge pull request kubernetes-sigs#117 from fengzixu/master 4ab8b15 use the tag to replace commit of csi-test 5d74e45 change the csi-test import path to v4 7dcd0a9 upgrade csi-test to v4.0.2 git-subtree-dir: release-tools git-subtree-split: 5489de6
humblec
pushed a commit
to humblec/nfs-subdir-external-provisioner
that referenced
this issue
May 11, 2022
…de-spellcheck-boilerplate-tests Add spelling and boilerplate checks
|
hello, Did you solve your problem? I have the same problem. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
version: v4.0.2
installation method: manual (no Helm chart)
K8s version: EKS 1.19
Issue:
the NFS provisioner claims to have created a volume in EFS:
I0707 16:38:11.043204 1 controller.go:1317] provision "default/test-claim" class "managed-nfs-storage": started
I0707 16:38:11.054723 1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-claim", UID:"bb4bd2b1-2237-4209-9894-b6ac135ace4c", APIVersion:"v1", ResourceVersion:"458642", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/test-claim"
I0707 16:38:11.054826 1 controller.go:1420] provision "default/test-claim" class "managed-nfs-storage": volume "pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c" provisioned
I0707 16:38:11.054846 1 controller.go:1437] provision "default/test-claim" class "managed-nfs-storage": succeeded
I0707 16:38:11.054850 1 volume_store.go:212] Trying to save persistentvolume "pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c"
I0707 16:38:11.337955 1 volume_store.go:219] persistentvolume "pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c" saved
I0707 16:38:11.338053 1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-claim", UID:"bb4bd2b1-2237-4209-9894-b6ac135ace4c", APIVersion:"v1", ResourceVersion:"458642", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c
however, the PV does not exist on the EFS file system and the pods cannot start because the volume is not found - kubelet logs:
Jul 7 16:42:25 ip-192-168-4-52 kubelet: I0707 16:42:25.397041 5268 reconciler.go:269] operationExecutor.MountVolume started for volume "pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c" (UniqueName: "kubernetes.io/nfs/8632c33d-37f7-464b-84ce-3796a13f862f-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c") pod "test-pod" (UID: "8632c33d-37f7-464b-84ce-3796a13f862f")
Jul 7 16:42:25 ip-192-168-4-52 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/8632c33d-37f7-464b-84ce-3796a13f862f/volumes/kubernetes.io
nfs/pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c.nfs/pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c --scope -- mount -t nfs -o hard,noresvport,retrans=2,rsize=1048576,timeo=600,vers=4.1,wsize=1048576 xxxx.efs.eu-west-1.amazonaws.com:/pvc/default-test-claim-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c /var/lib/kubelet/pods/8632c33d-37f7-464b-84ce-3796a13f862f/volumes/kubernetes.ioJul 7 16:42:25 ip-192-168-4-52 kubelet: E0707 16:42:25.439612 5268 mount_linux.go:150] Mount failed: exit status 32
Jul 7 16:42:25 ip-192-168-4-52 kubelet: Mounting command: systemd-run
Jul 7 16:42:25 ip-192-168-4-52 kubelet: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/8632c33d-37f7-464b-84ce-3796a13f862f/volumes/kubernetes.io
nfs/pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4cnfs/pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c --scope -- mount -t nfs -o hard,noresvport,retrans=2,rsize=1048576,timeo=600,vers=4.1,wsize=1048576 xxxxxx.efs.eu-west-1.amazonaws.com:/pvc/default-test-claim-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c /var/lib/kubelet/pods/8632c33d-37f7-464b-84ce-3796a13f862f/volumes/kubernetes.io~nfs/pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c\nOutput: Running scope as unit run-8385.scope.\nmount.nfs: mounting xxxxxx.efs.eu-west-1.amazonaws.com:/pvc/default-test-claim-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c failed, reason given by server: No such file or directory\n"Jul 7 16:42:25 ip-192-168-4-52 kubelet: Output: Running scope as unit run-8385.scope.
Jul 7 16:42:25 ip-192-168-4-52 kubelet: mount.nfs: mounting xxxx.efs.eu-west-1.amazonaws.com:/pvc/default-test-claim-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c failed, reason given by server: No such file or directory
Jul 7 16:42:25 ip-192-168-4-52 kubelet: E0707 16:42:25.440852 5268 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/nfs/8632c33d-37f7-464b-84ce-3796a13f862f-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c podName:8632c33d-37f7-464b-84ce-3796a13f862f nodeName:}" failed. No retries permitted until 2021-07-07 16:44:27.440818649 +0000 UTC m=+4259.482223844 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c" (UniqueName: "kubernetes.io/nfs/8632c33d-37f7-464b-84ce-3796a13f862f-pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c") pod "test-pod" (UID: "8632c33d-37f7-464b-84ce-3796a13f862f") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/8632c33d-37f7-464b-84ce-3796a13f862f/volumes/kubernetes.io
The PV/PVC are bound:
$ kubectl -n default get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Bound pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c 1Mi RWX managed-nfs-storage 20m
$ kubectl -n default get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
pvc-bb4bd2b1-2237-4209-9894-b6ac135ace4c 1Mi RWX Delete Bound default/test-claim managed-nfs-storage 6m46s
I can mount EFS on the EKS node successfully, but the dynamic provisioning of an EFS volume does not appear to work. Any hints would be much appreciated.
The text was updated successfully, but these errors were encountered: