Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixbug: check driver timeout #68

Merged
merged 1 commit into from
Sep 17, 2018

Conversation

wnxn
Copy link
Contributor

@wnxn wnxn commented Aug 24, 2018

Please reference Issue #67.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 24, 2018
@wnxn
Copy link
Contributor Author

wnxn commented Aug 25, 2018

/assign @jsafrane

Copy link
Contributor

@jsafrane jsafrane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am open to increasing probe timeout, however, I don't like the other changes.

@@ -120,7 +120,9 @@ func main() {
os.Exit(1)
}

supportsService, err := csiConn.SupportsPluginControllerService(ctx)
ctxController, cancelController := context.WithTimeout(context.Background(), csiTimeout)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like 3 different contexts. If you think that 1 second is too short for GetDriverName+SupportsPluginControllerService+SupportsControllerPublish together, then we could increase the context timeout to 3 seconds instead of using three different ones.

Copy link
Contributor Author

@wnxn wnxn Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @jsafrane
Thanks for your reply.
After only increasing Probe timeout, I still meet timeout error in external attacher container. We may not call waitForDriverReady during the three calls, GetDriverName+SupportsPluginControllerService+SupportsControllerPublish.
If we don't use 3 different contexts, we can call waitForDriverReady before or after
the three calls.
When calling waitForDriverReady before the three calls, may waste some time because probe is not a short call.
On the contrary, we could call waitForDriverReady after the three calls and before launching CSI handler or trivial handler.

only increasing Probe timeout in external attacher:

# kubectl logs csi-neonsan-controller-0 csi-attacher
I0827 07:58:09.721682       1 main.go:75] Version: v0.2.0-35-g5a4cc3b-dirty
I0827 07:58:09.727437       1 connection.go:88] Connecting to /var/lib/csi/sockets/pluginproxy/csi.sock
I0827 07:58:09.727923       1 connection.go:115] Still trying, connection is CONNECTING
I0827 07:58:09.728326       1 connection.go:112] Connected
I0827 07:58:09.728351       1 connection.go:235] GRPC call: /csi.v0.Identity/GetPluginInfo
I0827 07:58:09.728363       1 connection.go:236] GRPC request: 
I0827 07:58:09.731362       1 connection.go:238] GRPC response: name:"csi-neonsan" vendor_version:"0.3.0" 
I0827 07:58:09.731484       1 connection.go:239] GRPC error: <nil>
I0827 07:58:09.731495       1 main.go:115] CSI driver name: "csi-neonsan"
I0827 07:58:09.731585       1 connection.go:235] GRPC call: /csi.v0.Identity/Probe
I0827 07:58:09.731629       1 connection.go:236] GRPC request: 
I0827 07:58:11.783301       1 connection.go:238] GRPC response: ready:<value:true > 
I0827 07:58:11.783424       1 connection.go:239] GRPC error: <nil>
I0827 07:58:11.783439       1 main.go:202] Probe succeeded
I0827 07:58:11.783453       1 connection.go:235] GRPC call: /csi.v0.Identity/GetPluginCapabilities
I0827 07:58:11.783464       1 connection.go:236] GRPC request: 
I0827 07:58:11.784170       1 connection.go:238] GRPC response: 
I0827 07:58:11.784301       1 connection.go:239] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0827 07:58:11.784501       1 main.go:125] rpc error: code = DeadlineExceeded desc = context deadline exceeded

@wnxn wnxn force-pushed the bugfix/check_driver branch from 5956470 to 1ce827f Compare August 28, 2018 01:43
@@ -114,18 +114,17 @@ func main() {
}
glog.V(2).Infof("CSI driver name: %q", attacher)

// Check it's ready
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I see what you meant. Can you move this check before ctx, cancel := context.WithTimeout(context.Background(), csiTimeout) ~10 lines above?

I think the order of waitForDriverReady() and GetDriverName() does not really matter, so let's do probe first and then start 1 context for GetDriverName+SupportsPluginControllerService+SupportsControllerPublish.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @jsafrane
Currently, I move the probe check before GetDriverName+SupportsPluginControllerService+SupportsControllerPublish and increase probe timeout. Would you please review this PR again?

@wnxn wnxn force-pushed the bugfix/check_driver branch from 1ce827f to 599ca57 Compare August 28, 2018 08:29
@jsafrane
Copy link
Contributor

/lgtm
Thanks for the fix!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 28, 2018
@jsafrane
Copy link
Contributor

/approve
?

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane, wnxn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 17, 2018
@k8s-ci-robot k8s-ci-robot merged commit cf1ef8e into kubernetes-csi:master Sep 17, 2018
@jianglingxia
Copy link

@ jsafrane
Excuse me,in k8sv1.13 i create a pod with cinder-csi-plugin,the pod log is :rpc error: code = DeadlineExceeded desc = context deadline exceeded

Events:
Type Reason Age From Message


Normal Scheduled 1m default-scheduler Successfully assigned default/csi-pod-nodeworker to 111.111.111.116
Warning FailedAttachVolume 33s (x6 over 49s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-13dc8ae1-61c8-11e9-b4f9-3a8d7c0bb20e" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Normal SuccessfulAttachVolume 17s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-13dc8ae1-61c8-11e9-b4f9-3a8d7c0bb20e"

the log print :rpc error: code = DeadlineExceeded desc = context deadline exceeded is that within 15s
not attached the node? this is a problem ?and how to fix it ? thanks very much!

dobsonj pushed a commit to dobsonj/external-attacher that referenced this pull request Jan 17, 2024
STOR-1688: Chore: add .snyk file to ignore false positives
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants