Deleting an age-based Bigtable GCPolicy for a replicated cluster with kubectl hangs

[fosky94]

Deleting an age-based Bigtable GCPolicy for a replicated cluster with kubectl hangs instead of throwing and error.

To reproduce this issue, you need to declaratively create an age-based GCPolicy for a Bigtable Instance with >1 clusters. When deleting the GCPolicy it stays in a deleted state and the command hangs forever. Due to the prevention of deleting age-based garbage collection policies this should fail and throw an error, instead of stating that it deleted the GCPolicy and hang.

An example of how the cbt cli displays this error can be found below:

$ cbt setgcpolicy <REDACTED> <REDACTED> maxage=11d
Setting GC policy: rpc error: code = FailedPrecondition desc = Cannot relax pure age-based GC for a replicated family (<REDACTED>). If you must relax the age constraint, unreplicate the instance and try again.

Steps to reproduce this issue:

1. Create an instance with 2 clusters using the .yaml file attached at the bottom of this issue.

   $ kubectl -n<REDACTED> apply -f <file>.yaml

2. (optional) Check DI status:

   $ kubectl -n<REDACTED> get BigtableInstances <REDACTED>

3. (optional) Run cbt to check GCPolicies (note might have to wait awhile for everything to take effect):

   $ cbt -project <REDACTED> -instance <REDACTED> ls <REDACTED>
   Family Name	GC Policy
   -----------	---------
   onefam1		age() > 1h
   onefam2		<never>
   

4. Delete the gc policy

   $ kubectl -n<REDACTED> delete bigtablegcpolicy onefam1
   bigtablegcpolicy.bigtable.cnrm.cloud.google.com "<REDACTED>.one.onefam1" deleted
   hangs......
   

5. crtl-C the hanging command and get the status of the gcpolicy

   $ kubectl -n<REDACTED> get BigtableGCPolicies <REDACTED>.one.onefam1
   NAME                    AGE     READY   STATUS     STATUS AGE
   <REDACTED>.one.onefam1   6m45s   False   Deleting   2m45s
   

6. As expected rerunning cbt shows the policy still there:

   $ cbt -project <REDACTED> -instance <REDACTED> ls <REDACTED>
   Family Name	GC Policy
   -----------	---------
   onefam1		age() > 1h
   onefam2		<never>
   

if you edit the resource and remove the finalizer then the deletion completes, but cbt still shows the policy. So deletion wasn't successful.

Request:
Could it be possible to return an error similar to the one cbt displays?

Thank you in advance! :)

YAML file:

---
apiVersion: "bigtable.cnrm.cloud.google.com/v1beta1"
kind: "BigtableInstance"
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: "REDACTED"
    cnrm.cloud.google.com/deletion-policy: abandon
  name: "REDACTED"
spec:
  displayName: "REDACTED"
  instanceType: "PRODUCTION"
  resourceID: "REDACTED"
  cluster:
  - clusterId: "REDACTED-c1"
    zone: "europe-west1-b"
    storageType: "HDD"
  - clusterId: "REDACTED-c2"
    zone: "europe-west1-c"
    storageType: "HDD"
---
apiVersion: "bigtable.cnrm.cloud.google.com/v1beta1"
kind: "BigtableTable"
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: "REDACTED"
    cnrm.cloud.google.com/deletion-policy: abandon
  name: "REDACTED.one"
spec:
  columnFamily:
  - family: "onefam1"
  - family: "onefam2"
  instanceRef:
    external: "REDACTED"
  resourceID: "one"
---
apiVersion: "bigtable.cnrm.cloud.google.com/v1beta1"
kind: "BigtableGCPolicy"
metadata:
  name: "REDACTED.one.onefam1"
spec:
  columnFamily: "onefam1"
  instanceRef:
    name: "REDACTED"
  maxAge:
  - duration: "3600s"
  tableRef:
    name: "REDACTED.one"
---
apiVersion: "bigtable.cnrm.cloud.google.com/v1beta1"
kind: "BigtableGCPolicy"
metadata:
  name: "REDACTED.one.onefam2"
spec:
  columnFamily: "onefam2"
  instanceRef:
    name: "REDACTED"
  tableRef:
    name: "REDACTED.one"

[xiaobaitusi]

Hi @fosky94, sorry for the late reply.

I have been able to reproduce the issue, it seems that the hanging operation is buried in the terraform implementation of google_bigtable_gc_policy, it has been continuously retrying on the following error for some reason. I'll dig a little deep and circle back.

google_bigtable_gc_policy.policy: Still destroying... [id=age() > 2h, 50s elapsed]
2021-09-21T23:16:52.318-0700 [INFO]  plugin.terraform-provider-google_v3.85.0_x5: 2021/09/21 23:16:52 [DEBUG] Dismissed an error as retryable. Waiting for table to be in a valid state - rpc error: code = FailedPrecondition desc = Cannot relax pure age-based GC for a replicated family (onefam3). If you must relax the age constraint, unreplicate the instance and try again.: timestamp=2021-09-21T23:16:52.318-0700
2021-09-21T23:16:52.318-0700 [INFO]  plugin.terraform-provider-google_v3.85.0_x5: 2021/09/21 23:16:52 [TRACE] Waiting 10s before next try: timestamp=2021-09-21T23:16:52.318-0700
google_bigtable_gc_policy.policy: Still destroying... [id=age() > 2h, 1m0s elapsed]
2021-09-21T23:17:02.993-0700 [INFO]  plugin.terraform-provider-google_v3.85.0_x5: 2021/09/21 23:17:02 [DEBUG] Dismissed an error as retryable. Waiting for table to be in a valid state - rpc error: code = FailedPrecondition desc = Cannot relax pure age-based GC for a replicated family (onefam3). If you must relax the age constraint, unreplicate the instance and try again.: timestamp=2021-09-21T23:17:02.992-0700
2021-09-21T23:17:02.993-0700 [INFO]  plugin.terraform-provider-google_v3.85.0_x5: 2021/09/21 23:17:02 [TRACE] Waiting 10s before next try: timestamp=2021-09-21T23:17:02.992-0700
...

[xiaobaitusi]

File an issue in terraform for inquiry and tracking: hashicorp/terraform-provider-google#10132

[kevinsi4508]

I believe this issue should be closed once we updated KCC to use the latest TF provider. See hashicorp/terraform-provider-google#10132.

[mbzomowski]

spec.gcRules has been added to BigTableGCPolicy as of Config Connector v1.97.0. We don't recommend using maxAge, maxVersion or mode for defining a
BigTableGCPolicy as these fields have known drift detection issues. Please use gcRules instead.