Conformance tests failing with projectcalico release: already exists

[willie-yao]

Which tests are failing:
Various conformance tests

Since when has it been failing:

Testgrid link:
Example: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/4155/pull-cluster-api-provider-azure-conformance-ipv6-with-ci-artifacts/1725228044999725056

Reason for failure (if possible):
Possibly a regression with CAAPH

Anything else we need to know:

/kind failing-test

[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]