Skip to content

Commit

Permalink
server: Target the spec configuration if we have at least one node
Browse files Browse the repository at this point in the history
The CI cluster hit an issue where a pull secret was broken, and
then we hit a deadlock because the MCO failed to drain nodes on
the old config, because other nodes on the old config couldn't
schedule the pod.

It just generally makes sense for new nodes to use the new config;
do so as long as at least one node has successfully joined the
cluster at that config.  This way we still avoid breaking
the cluster (and scaleup) with a bad config.
  • Loading branch information
cgwalters committed Aug 27, 2020
1 parent 36f37f2 commit 4bd204d
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion pkg/server/cluster_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,16 @@ func (cs *clusterServer) GetConfig(cr poolRequest) (*runtime.RawExtension, error
return nil, fmt.Errorf("could not fetch pool. err: %v", err)
}

currConf := mp.Status.Configuration.Name
// For new nodes, we roll out the latest if at least one node has successfully updated.
// This avoids deadlocks in situations where the old configuration broke somehow
// (e.g. pull secret expired)
// and also avoids provisioning a new node, only to update it not long thereafter.
var currConf string
if mp.Status.UpdatedMachineCount > 0 {
currConf = mp.Spec.Configuration.Name
} else {
currConf = mp.Status.Configuration.Name
}

mc, err := cs.machineClient.MachineConfigs().Get(context.TODO(), currConf, metav1.GetOptions{})
if err != nil {
Expand Down

0 comments on commit 4bd204d

Please sign in to comment.