[Bug]: Crossplane repeatedly tries to modify already existing resource when no changes are being made

[artisticcheese]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

cdn.azure.upbound.io/v1beta1/FrontdoorCustomDomain

Resource MRs required to reproduce the bug

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorCustomDomain
metadata:
  name: product1-customdomain1
spec:
  providerConfigRef:
    name: workload-identity-provider-config
  forProvider:
    cdnFrontdoorProfileIdRef:
      name: frontdoor-reference
    dnsZoneId: "/subscriptions/11b16e25-87bc-5f2e9adb26df/resourceGroups/core-infra-eastus-dnszones/providers/Microsoft.Network/dnszones/cdn.dev1.intapp.com"
    hostName: product1-customdns1.cdn.dev1.intapp.com
    tls:
      - certificateType: ManagedCertificate
        minimumTlsVersion: TLS12

Steps to Reproduce

Applied resource

What happened?

CrossPlane consistently detects changes in resource even thought there are no changes based on logs in crossplane as well as in Azure

Relevant Error Output Snippet

2024-12-16T22:41:19Z    DEBUG   provider-azure  Reconciling     {"controller": "managed/cdn.azure.upbound.io/v1beta1, kind=frontdoorcustomdomain", "request": {"name":"product1-customdomain1"}}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Connecting to the service provider      {"uid": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "name": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Observing the external resource {"uid": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "name": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Diff detected   {"uid": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "name": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"dns_zone_id\":*terraform.ResourceAttrDiff{Old:\"/subscriptions/XXXX/resourceGroups/core-infra-eastus-dnszones/providers/Microsoft.Network/dnsZones/cdn.dev1.intapp.com\", New:\"/subscriptions/XXXX/resourceGroups/core-infra-eastus-dnszones/providers/Microsoft.Network/dnszones/cdn.dev1.intapp.com\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Successfully requested update of external resource      {"controller": "managed/cdn.azure.upbound.io/v1beta1, kind=frontdoorcustomdomain", "request": {"name":"product1-customdomain1"}, "uid": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "version": "1692595947", "external-name": "product1-customdomain1", "requeue-after": "2024-12-16T22:51:38Z"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Async update starting...        {"trackerUID": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "resourceName": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain", "tfID": "/subscriptions/XXXX/resourceGroups/aso-greg-test-rg/providers/Microsoft.Cdn/profiles/crossplane-frontdoor/customDomains/product1-customdomain1"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Updating the external resource  {"uid": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "name": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain"}
2024-12-16T22:41:19Z    DEBUG   provider-azure  Async update ended.     {"trackerUID": "9be4f79e-3c97-4d43-9341-c0d7a364fe6c", "resourceName": "product1-customdomain1", "gvk": "cdn.azure.upbound.io/v1beta1, Kind=FrontdoorCustomDomain", "error": null, "tfID": "/subscriptions/XXXX/resourceGroups/aso-greg-test-rg/providers/Microsoft.Cdn/profiles/crossplane-frontdoor/customDomains/product1-customdomain1"}

Crossplane Version

1.18

Provider Version

1.10

Kubernetes Version

1.31.2

Kubernetes Distribution

AKS

Additional Info

It seems to be trying update property azureDnsZone based on activity log, even though this property never changed, it succeeds and then tries to redo it again and again.

"requestbody": "{\"properties\":{\"azureDnsZone\":{\"id\":\"/subscriptions/xxx/resourceGroups/core-infra-eastus-dnszones/providers/Microsoft.Network/dnszones/cdn.dev1.intapp.com\"}}}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/xxx/resourceGroups/aso-greg-test-rg/providers/Microsoft.Cdn/profiles/crossplane-frontdoor/customDomains/product1-customdomain1",
        "message": "Microsoft.Cdn/profiles/customDomains/write",

[naimadswdn]

I faced the same behavior for the KubernetesCluster and ApplicationGateway resources.
For the KubernetesCluster, the issue was because the provider detected drift for the api_server_access_profile (url) that I event didn't have defined in my MR definition.
I fixed it by adding:

  apiVersion: containerservice.azure.upbound.io/v1beta1
  kind: KubernetesCluster
  spec:
    forProvider:
      apiServerAccessProfile: []

When it comes to the ApplicationGateway the problem was with the probe block. I was missing match and minimumServers values there and because of that, the network provider was detecting the drift.

NOTE: What is interesting I tried to use both the managementPolicies without the 'LateInitialize' and the initProvider but both were not helping.

All these happen after the providers update from the version v1.5.1 to v1.11.0.

So lesion for me and for others is to double check the MRs after providers update and always use the DEBUG provider config to see what is the actual problem.

When it comes to the issue raised by the @artisticcheese , I also use the FrontdoorCustomDomain in my code base, but with BYO DNS, so not using the Azure DNS.

Is the dnsZoneId provided as a hardcoded string or it is passed during the composition reconciliation?