Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The PV could created in a new deployed csi-driver, more than 1.5 hour later, the pv couldn't be cerated with timeout error #246

Closed
darcyllingyan opened this issue Mar 12, 2019 · 4 comments
Closed

The PV could created in a new deployed csi-driver, more than 1.5 hour later, the pv couldn't be cerated with timeout error #246

darcyllingyan opened this issue Mar 12, 2019 · 4 comments

Comments

@darcyllingyan
Copy link

darcyllingyan commented Mar 12, 2019 •
edited
Loading

Hi,
I want to create a POD with the volume provided by CSI, it failed as the volume creating process.
The log is below:
1.

# kubectl get pod
NAME                                   READY   STATUS                       RESTARTS   AGE
hardy-antelope-cnet-7b6d4b99c6-k8lhs   0/1     Pending                      0          22m

# kubectl describe pod hardy-antelope-cnet-7b6d4b99c6-k8lhs
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  13s (x3 over 13s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 4 times)

# kubectl describe pvc hardy-antelope-cnet-db
Events:
  Type       Reason                Age               From                                                                                  Message
  ----       ------                ----              ----                                                                                  -------
  Normal     Provisioning          29s               csi-cinderplugin_csi-provisioner-cinderplugin-0_a9093f59-440a-11e9-8e03-9ab79fafed29  External provisioner is provisioning volume for claim "default/hardy-antelope-cnet-db"
  Normal     ExternalProvisioning  2s (x4 over 29s)  persistentvolume-controller                                                           waiting for a volume to be created, either by external provisioner "csi-cinderplugin" or manually created by system administrator
Mounted By:  hardy-antelope-cnet-7b6d4b99c6-k8lhs

# docker ps|grep csi-provi
e75b70c484f3        bcmt-registry:5000/cinder-csi-plugin@sha256:ae6be6d8c3c27eac12580c02f6943c2ae9d588e50da6c64c2701ffed6edef164                           "/bin/cinder-csi-p..."   17 hours ago        Up 17 hours                             k8s_cinder_csi-provisioner-cinderplugin-0_kube-system_9c2b4136-440a-11e9-8ef3-fa163edde5cd_0
0720642fd468        bcmt-registry:5000/csi-provisioner@sha256:b83d89512e7a8b2e7790b107d72ac2260a92b84df3a7ad4a4dd3fd1065c8c140                             "/csi-provisioner ..."   17 hours ago        Up 17 hours                             k8s_csi-provisioner_csi-provisioner-cinderplugin-0_kube-system_9c2b4136-440a-11e9-8ef3-fa163edde5cd_0

# docker logs 0720642fd468

W0312 07:38:30.690760       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:38:30.691041       1 controller.go:685] Retrying syncing claim "default/hardy-antelope-cnet-db" because failures 2 < threshold 15
E0312 07:38:30.691134       1 controller.go:700] error syncing claim "default/hardy-antelope-cnet-db": failed to provision volume with StorageClass "csi-sc-cinderplugin": timed out waiting for the condition
I0312 07:38:30.691254       1 controller.go:926] provision "default/hardy-antelope-cnet-db" class "csi-sc-cinderplugin": started
I0312 07:38:30.691930       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"hardy-antelope-cnet-db", UID:"682f1e37-4498-11e9-87cf-fa163ed94ef7", APIVersion:"v1", ResourceVersion:"3973701", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-sc-cinderplugin": timed out waiting for the condition
I0312 07:38:30.710870       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"hardy-antelope-cnet-db", UID:"682f1e37-4498-11e9-87cf-fa163ed94ef7", APIVersion:"v1", ResourceVersion:"3973701", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/hardy-antelope-cnet-db"
W0312 07:38:40.712743       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:38:55.713479       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:39:11.714051       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:39:28.914482       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:39:47.554839       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:40:07.923777       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried
W0312 07:40:30.366973       1 controller.go:592] CreateVolume timeout: 10s has expired, operation will be retried

note: In the cluster, I configure both the cinder csi and the in-tree methond to provide cinder volume.

# kubectl get sc
NAME                       PROVISIONER                    AGE
cinder                     kubernetes.io/cinder           47h
cinder-az-nova (default)   kubernetes.io/cinder           47h
cinder-az-nova-xfs         kubernetes.io/cinder           47h
cinder-xfs                 kubernetes.io/cinder           47h
csi-sc-cinderplugin        csi-cinderplugin               47h
csi-sc-cinderplugin-nova   csi-cinderplugin               115m
local-storage              kubernetes.io/no-provisioner   47h

Can you help to check it about the root cause? I have no idea about it after checking several days, thanks very much!

THanks
@darcyllingyan
Copy link
Author

Does anyone has any suggestions? Thanks.

@darcyllingyan darcyllingyan changed the title Create volume timeout with the external provisioner The PV could created in a new created cluster, more than one hour later, the pv couldn't be cerated with timeout error Mar 17, 2019
@darcyllingyan darcyllingyan changed the title The PV could created in a new created cluster, more than one hour later, the pv couldn't be cerated with timeout error The PV could created in a new deployed csi-driver, more than one hour later, the pv couldn't be cerated with timeout error Mar 17, 2019
@darcyllingyan darcyllingyan changed the title The PV could created in a new deployed csi-driver, more than one hour later, the pv couldn't be cerated with timeout error The PV could created in a new deployed csi-driver, more than 1.5 hour later, the pv couldn't be cerated with timeout error Mar 17, 2019
@msau42
Copy link
Collaborator

msau42 commented Mar 17, 2019

Hi, I would suggest opening an issue with the cinder experts: https://github.com/kubernetes/cloud-provider-openstack/tree/master/pkg/csi/cinder

@darcyllingyan
Copy link
Author

Hi @msau42 ,
Thanks very much for the information, I have opened an isue kubernetes/cloud-provider-openstack#535, and still waiting for the response. Thanks.

@darcyllingyan
Copy link
Author

The issue is duplicate with #535, so close the issue.

andyzhangx added a commit to andyzhangx/external-provisioner that referenced this issue Dec 16, 2023
@andyzhangx
Squashed 'release-tools/' changes from f8c8cc4c7..b54c1ba49
c2cae10 
b54c1ba49 Merge pull request kubernetes-csi#246 from xing-yang/go_1.21
5436c81e9 Change go version to 1.21.5
267b40e97 Merge pull request kubernetes-csi#244 from carlory/sig-storage
b42e5a2de nominate self (carlory) as kubernetes-csi reviewer
a17f536fc Merge pull request kubernetes-csi#210 from sunnylovestiramisu/sidecar
011033de2 Use set -x instead of die
5deaf667c Add wrapper script for sidecar release

git-subtree-dir: release-tools
git-subtree-split: b54c1ba49469d4d5d1b5d75285e8868ffe3d328f
kbsonlong pushed a commit to kbsonlong/external-provisioner that referenced this issue Dec 29, 2023
@k8s-ci-robot
Merge pull request kubernetes-csi#246 from pohly/prow-update-master
a887c14 
master: update release-tools
tyuchn added a commit to tyuchn/external-provisioner that referenced this issue Mar 19, 2024
@tyuchn
Squashed 'release-tools/' changes from de2fba88b..dc4d0ae20
d1a4ca6 
dc4d0ae20 Merge pull request kubernetes-csi#249 from jsafrane/use-go-version
e681b170e Use .go-version to get Kubernetes go version
b54c1ba49 Merge pull request kubernetes-csi#246 from xing-yang/go_1.21
5436c81e9 Change go version to 1.21.5
267b40e97 Merge pull request kubernetes-csi#244 from carlory/sig-storage
b42e5a2de nominate self (carlory) as kubernetes-csi reviewer
a17f536fc Merge pull request kubernetes-csi#210 from sunnylovestiramisu/sidecar
011033de2 Use set -x instead of die
5deaf667c Add wrapper script for sidecar release
f8c8cc4c7 Merge pull request kubernetes-csi#237 from msau42/prow
b36b5bfdc Merge pull request kubernetes-csi#240 from dannawang0221/upgrade-go-version
adfddcc9a Merge pull request kubernetes-csi#243 from pohly/git-subtree-pull-fix
c4650889d pull-test.sh: avoid "git subtree pull" error
7b175a1e2 Update csi-test version to v5.2.0
987c90ccd Update go version to 1.21 to match k/k
2c625d41d Add script to generate patch release notes
f9d5b9c05 Merge pull request kubernetes-csi#236 from mowangdk/feature/bump_csi-driver-host-path_version
b01fd5372 Bump csi-driver-host-path version up to v1.12.0
984feece4 Merge pull request kubernetes-csi#234 from siddhikhapare/csi-tools
1f7e60599 fixed broken links of testgrid dashboard

git-subtree-dir: release-tools
git-subtree-split: dc4d0ae20a3dcce17fbfc745fb1f1e3b10cd9644
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@msau42 @darcyllingyan