Not possible to migrate existing AzureCluster with empty subscriptionID to CAPZ v1.11 or newer, which removes the fallback credential

[dlipovetsky]

/kind bug

What steps did you take and what happened:

1. I deployed CAPZ v1.10.5, and provided a fallback credential by defining environment variables in the manager container.
   (Note that this is done by setting values in the manager-bootstrap-credentials Secret that is then referenced by the container. This is in the manager Deployment, from the patch defined here.)
2. I created an AzureCluster that has an empty .spec.subscriptionID. CAPZ reconciled this cluster, using its fallback credential for the identity (tenant ID, client ID, client secret), and the subscription ID.
3. I upgraded CAPZ to v1.11.0. This is the first release to remove the manager-bootstrap-credentials patch, and to require the use of AzureClusterIdentity, both changes introduced by Require AzureClusterIdentity for auth #3793. CAPZ failed to reconcile my AzureCluster, because it did not have a valid identityRef.
   I created a Secret, a AzureClusterIdentity, and updated the identityRef. CAPZ still failed to reconcile my AzureCluster, because the manager did not know the subscription ID. This is because the AzureCluster spec.subscriptionID is empty, and the manager no longer uses fallback credentials.
   I tried to fix this by setting the AzureCluster spec.subscriptionID to the right ID, but I could not, because the field is immutable. I then had to modify the manager Deployment to inject the subscription ID.

What did you expect to happen:

I expected CAPZ v1.11.0 to allow me to update my AzureCluster spec.subscriptionID, so that it could reconcile it without needing the manager needing a fallback credential, because the fallback credential is no longer supported.

Anything else you would like to add:

• Today, spec.subscriptionID is an optional field. Arguably, it should be a required field for AzureCluster, as well as AzureManagedControlPlane and AzureManagedControlPlaneTemplate. For all these resources, the field is immutable. If we wanted to require the field, we would have to make it mutable, either write-many, or write-once.

• I first discussed this issue in the CAPZ slack channel.

Environment:

• cluster-api-provider-azure version: v1.10.5 and v1.11.0
• Kubernetes version: (use kubectl version): v1.27.6
• OS (e.g. from /etc/os-release): n/a