Pod volume restores always run after all data mover restores completes for the same pod

[Lyndon-Li]

As the scenario of #6658, when we have both PVR and data mover in the restore of the same pod, below behaviors will always happen:

1. PVRs won't start unless all the data mover restores completes
2. If one or more data mover restores fail or being cancelled, none of the PVRs will start

The reason is:

• PVRs won't start when the init-container of the pod runs
• The init-container won't run unless the storage of the pod is ready, or in the other word, all the PVCs are bound
• The PVCs to be restored by data movers won't be bound until the data mover restore is ready

[Lyndon-Li]

Why PVR ensures the init-container is running:

• This is an indirect way to check if all the volumes are provisioned, otherwise, if volumes are not provisioned, PVRs will fail
• PVRs need to ensure the pod is not running all the time during the restore, otherwise, the restore and the application may do conflict operations

We need to review and refactor this init-container usage to fix the problem.

[reasonerjt]

@Lyndon-Li We probably can customize the behavior of the init-container, such as passing the volume list as a parm to the process?

[Lyndon-Li]

After further investigation, we found:

• Pod's volumes are not attached until all the PVCs getting into Bound status
• However, the PVCs restored by data mover won't turn to Bound until the corresponding DDCRs complete
• As a result, none of the volumes are attached until data mover finishes, even though the PVCs that are not handled by data mover have been in Bound status
• Since volumes are not attached, PVRs have no way to start since the volumes are not in the host at all

Therefore, given the current behavior of PVRs (needs to visit the restored workload pods' volume) and the current behavior of Kuberentes' provisioner, there is not a way to solve this problem.

[Lyndon-Li]

Let's add the scenarios to the fs-backup document.

[sseago]

@Lyndon-Li I think that is fine. We should document it, but it's not a big deal. I suspect this is a bit of an edge case. Most "mixed datamover/fs-backup" scenarios will probably be some pods with datamover volumes, and others with fs backup volumes.