Service CRDs fail to be deleted when underlying GCP Project no longer exists

[lostick]

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

I am unable to delete Service CRDs when the GCP Project referred in the CRD has already been shut down

Additional Diagnostic Information

Kubernetes Cluster Version

v1.21.5-gke.1802

Config Connector Version

1.72.1

Config Connector Mode

cluster

Log Output

{"severity":"error","timestamp":"2022-02-09T10:34:21.202Z","logger":"controller.service-controller","msg":"Reconciler error","reconciler group":"serviceusage.cnrm.cloud.google.com","reconciler kind":"Service","name":"<PROJECT_ID>-dataproc-service","namespace":"tenant","error":"Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Project Service projects/<PROJECT_ID>/services/: Request `List Project Services <PROJECT_ID>` returned error: Failed to list enabled services for project <PROJECT_ID>: googleapi: Error 403: Project 'projects/<PROJECT_ID>' not found or permission denied. Details:\n[\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.PreconditionFailure\",\n    \"violations\": [\n      {\n        \"subject\": \"?error_code=210002\\u0026type=Project\\u0026resource_id=projects/<PROJECT_ID>\",\n        \"type\": \"googleapis.com\"\n      }\n    ]\n  },\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n    \"domain\": \"serviceusage.googleapis.com\",\n    \"metadata\": {\n      \"resource_id\": \"projects/<PROJECT_ID>\",\n      \"type\": \"Project\"\n    },\n    \"reason\": \"RESOURCES_NOT_FOUND\"\n  }\n]\n, forbidden"}

YAML snippets

apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
kind: Service
...
status:
  conditions:
  - lastTransitionTime: "2021-12-10T13:36:59Z"
    message: |-
      Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Project Service projects/<PROJECT_ID>/services/: Request `List Project Services <PROJECT_ID>` returned error: Failed to list enabled services for project <PROJECT_ID>: googleapi: Error 403: Project 'projects/<PROJECT_ID>' not found or permission denied.
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
          "violations": [
            {
              "subject": "?error_code=210002\u0026type=Project\u0026resource_id=projects/<PROJECT_ID>",
              "type": "googleapis.com"
            }
          ]
        },
        {
          "@type": "type.googleapis.com/google.rpc.ErrorInfo",
          "domain": "serviceusage.googleapis.com",
          "metadata": {
            "resource_id": "projects/<PROJECT_ID>",
            "type": "Project"
          },
          "reason": "RESOURCES_NOT_FOUND"
        }
      ]
      , forbidden
    reason: DeleteFailed
    status: "False"
    type: Ready
  observedGeneration: 2

[xiaobaitusi]

Hi @lostick, this is sort of WAI. The service controller consumes that the associated project is active when enabling and disabling individual service.

Is the associated project also managed by Config Connector? If so, you can try to reference to the project with projectRef.name field in the Service CRD and delete them together. When Config Connector finds the referenced project is removed from the k8s cluster, it will orphan the Service object.

Also, you might be able to apply cnrm.cloud.google.com/deletion-policy: "abandon" annotation into the Service object, and allow Config Connector to abandon the Service, i.e. perform no-ops against the GCP API on deletion. Because the underlying project is deleted, it should be safe to just abandon the k8s objects.

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  annotations:
    cnrm.cloud.google.com/auto-create-network: "false"
  labels:
    label-one: "value-one"
  name: my-project
spec:
  name: Config Connector Sample
  folderRef:
    # Replace "${FOLDER_ID?}" with the numeric ID of the parent folder
    external: "${FOLDER_ID?}"
  billingAccountRef:
    # Replace "${BILLING_ACCOUNT_ID?}" with the numeric ID for your billing account
    external: "${BILLING_ACCOUNT_ID?}"
---
apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
kind: Service
metadata:
  annotations:
    # use the deletion policy of abandon to ensure that the pubsub service remains enabled when this resource is deleted.
    cnrm.cloud.google.com/deletion-policy: "abandon"
    # this is unnecessary with the deletion-policy of 'abandon', but useful if the abandon policy is removed.
    cnrm.cloud.google.com/disable-dependent-services: "false"
  name: pubsub.googleapis.com
spec:
    projectRef: 
      name: my-project

Let us know if those two alternatives work for you.

[lostick]

thanks @xiaobaitusi, with the abandon annotation we can delete the CRD