Got error: Node \"node02\" is invalid: spec.providerID: Forbidden: node updates may not change providerID except from \"\" to valid

[huchen2021]

I met a bug when I do test “Create a workload cluster with 3 cp + 3md”. This test can passed with a clear environment. But in my environment, it reused a lot of times, and trigger this following issue:

The second control plane entering “Provisioning” status forever.

I met the following error in byoh-controller-manager.log:

E1216 08:50:09.084241 1 byomachine_controller.go:272] controller/byomachine "msg"="failed to set node providerID" "error"="Node "node02" is invalid: spec.providerID: Forbidden: node updates may not change providerID except from "" to valid" "cluster"="byoh-wc" "name"="byoh-wc-control-plane-29fxm" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
E1216 08:50:09.085312 1 controller.go:317] controller/byomachine "msg"="Reconciler error" "error"="Node "node02" is invalid: spec.providerID: Forbidden: node updates may not change providerID except from "" to valid" "name"="byoh-wc-control-plane-29fxm" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"

After read the code and analyse the log, I figure out the reason of this issue. This reason is the first time to set “Updating Node with ProviderID” failed with timeout, but actually the node object “node02" already update with spec.ProviderID “byoh://node02/hg7i3d”.

The content of byoh-controller-manager.log:

E1216 08:50:03.164717 1 byomachine_controller.go:272] controller/byomachine "msg"="failed to set node providerID" "error"="etcdserver: request timed out" "cluster"="byoh-wc" "name"="byoh-wc-control-plane-29fxm" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
E1216 08:50:03.165896 1 controller.go:317] controller/byomachine "msg"="Reconciler error" "error"="etcdserver: request timed out"

The yaml output of node "node02":

Then every time after that will be report this error in byoh-controller-manager**.log:

E1216 08:50:09.084241 1 byomachine_controller.go:272] controller/byomachine "msg"="failed to set node providerID" "error"="Node "node02" is invalid: spec.providerID: Forbidden: node updates may not change providerID except from "" to valid" "cluster"="byoh-wc" "name"="byoh-wc-control-plane-29fxm" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"
E1216 08:50:09.085312 1 controller.go:317] controller/byomachine "msg"="Reconciler error" "error"="Node "node02" is invalid: spec.providerID: Forbidden: node updates may not change providerID except from "" to valid" "name"="byoh-wc-control-plane-29fxm" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="ByoMachine"

And this error output is because it is not allowed to update the spec.providerID of node object. You can try to delete this field of a node object, then you’ll see this error output. With this error, the control plane entering “Provisioning” status forever.