Need the ability to control what nodes the data movers are scheduled on

[CrimsonFez]

Describe the problem/challenge you have

I want to be able to restrict the nodes that data mover pods can be scheduled on. This is similar to #6734. Correct me if I'm wrong, but the way data movement works is that the node agent daemonset handles data movement while the pod that is deployed for each PVC is just so that PVC is created and mounted somewhere.

The issue occurs when you don't have the node agent on all nodes, kubernetes will schedule the dummy pod on a node where the node agent isn't deployed causing the upload to stall.

Describe the solution you'd like

When deploying the dummy pods use the nodeSelector and affinity that is specified on the node agent daemonset.

Environment:

velero 1.12.2 with csi 0.6.2 and aws 1.8.2 plugins
debian 11
rke2 1.25.13

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[Lyndon-Li]

See this issue #7036, for backup I think this is what you are requesting.
However, for restore, things are somehow different --- if the volume's volumeBindingMode (defined by its storageclass) is waitForFirstConsumer, the data movement must run in the same node where the restored pod is running. In this case, there is no choice; on the other hand, if volumeBindingMode is immediate, we are free to chose node.

Therefore, let us know which part (backup/restore) you are concerning and whether #7036 could meet your requirements.

[Lyndon-Li]

Additionally, please share us why this happens, Velero node-agent is a daemonset, so theoretically it should be on every worker node:

you don't have the node agent on all nodes

[CrimsonFez]

Additionally, please share us why this happens, Velero node-agent is a daemonset, so theoretically it should be on every worker node:

you don't have the node agent on all nodes

I have scheduling allowed on my control plane, but I don't want data movers to be scheduled there.

See this issue #7036, for backup I think this is what you are requesting. However, for restore, things are somehow different --- if the volume's volumeBindingMode (defined by its storageclass) is waitForFirstConsumer, the data movement must run in the same node where the restored pod is running. In this case, there is no choice; on the other hand, if volumeBindingMode is immediate, we are free to chose node.

Therefore, let us know which part (backup/restore) you are concerning and whether #7036 could meet your requirements.

I've only thought of the backup process. When talking about the restore process, my solution wouldn't be adequate. In that case a whitelist/blacklist would be a better solution.

I would like to add, having a whitelist/blacklist for the backup process would be desirable for bare-metal clusters where you might have dedicated storage nodes. That way backup processes are performed on nodes with faster data and network access. Additionally, I think that the whitelist/blacklist shouldn't be a simple list, but instead some sort of label selector.

I'm going to close this in lieu of #7036