[WIP]fix windows node register failure issue #149
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,10 +24,12 @@ import ( | |
| "os/signal" | ||
| "runtime" | ||
| "syscall" | ||
| "time" | ||
|
|
||
| "google.golang.org/grpc" | ||
|
|
||
| "github.com/kubernetes-csi/node-driver-registrar/pkg/util" | ||
| "k8s.io/apimachinery/pkg/util/wait" | ||
| "k8s.io/klog/v2" | ||
| registerapi "k8s.io/kubelet/pkg/apis/pluginregistration/v1" | ||
| ) | ||
|
|
@@ -60,8 +62,25 @@ func nodeRegister(csiDriverName, httpEndpoint string) { | |
| } | ||
| klog.Infof("Registration Server started at: %s\n", socketPath) | ||
| grpcServer := grpc.NewServer() | ||
|
|
||
| // Registers kubelet plugin watcher api. | ||
| if runtime.GOOS == "windows" { | ||
| for true { | ||
|
There was a problem hiding this comment. Instead of retrying forever, do we want to return fatalf? There was a problem hiding this comment. try a few times(e.g. 10 times) is also an option, I am ok with that. |
||
| // try register on Windows node in a loop, detailed issue: | ||
| // https://github.com/kubernetes-csi/node-driver-registrar/issues/143 | ||
| err := wait.PollImmediate(1*time.Second, 30*time.Second, func() (bool, error) { | ||
| registerapi.RegisterRegistrationServer(grpcServer, registrar) | ||
|
There was a problem hiding this comment. The issue description says that this call hangs forever. I don't think this Poll method is going to ever return in that case. I think you'll need to run the call in a go routine and wait for the result or timeout that way. There was a problem hiding this comment. I think that's a wrong fix, do you know is there any way to restart node-driver-registrar automatically if register is not succeeded within a few seconds? The registration issue happens on both Azure & GKE Windows node now. When we hit the issue, restart node driver always works. There was a problem hiding this comment. I think returning a fatal error is the main way to cause node-driver-registrar to restart. There was a problem hiding this comment. From the log you showed that the hang happened after this registration process. So will loop here help? There was a problem hiding this comment. This PR won't work, after node-driver-registrar/cmd/csi-node-driver-registrar/main.go Lines 78 to 80 in 1c2d606 |
||
| return true, nil | ||
| }) | ||
| if err == nil { | ||
| break | ||
| } else if err == wait.ErrWaitTimeout { | ||
| klog.Errorf("RegisterRegistrationServer timeout, try again") | ||
| } | ||
| } | ||
| } else { | ||
| registerapi.RegisterRegistrationServer(grpcServer, registrar) | ||
| } | ||
|
|
||
| go healthzServer(socketPath, httpEndpoint) | ||
| go removeRegSocket(csiDriverName) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure that this is the line where it hangs? I checked the kubelet & gRPC codebase and I see that it's just setting some properties in the server but it's not making any calls:
https://github.com/kubernetes/kubelet/blob/v0.21.1/pkg/apis/pluginregistration/v1/api.pb.go#L213-L215
https://github.com/grpc/grpc-go/blob/0257c8657362b76f24e7a8cfb61df48d4cb735d3/server.go#L625-L661
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andyzhangx comment here #149 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
got it, thanks Jing, so it looks like after registration the kubelet doesn't call node-driver-registrar back (which could be because the request never got to the Kubelet or the kubelet acknowledged the request and didn't reply back), I think that if this is reproducible we should check the kubelet logs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mauriciopoppe I am not quite familiar with node-driver-registrar, is there any way to fail when node-driver-registrar did not receive ack from kubelet? There could be many issues that kubelet does not respond, if node-driver-registrar could fail, then it would retry, I think that would fix this issue.
This PR does not fix the issue actully.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add some comments to #143