Destroy resource group failed with 204

[ocofaigh]

According to the stdout from terraform destroy, the resource group deleted successfully:

Destroying... [id=4b1a9955550b4a59b1e9db20a39f7edf]
ibm_resource_group.test_resource_group: Destruction complete after 0s

But the terraform destroy failed with:

╷
│ Error: Unable to delete object
│ 
│ 
╵

Looking at the debug trace, the request returned a 204 (No content). Why is is getting a 204? Is this a timing issue? Was the group already destroyed and a retry attempt got a 204?

2022-01-26T06:28:57.678Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:57 [Debug] Request:
DELETE /v2/resource_groups/4b1a9955550b4a59b1e9db20a39f7edf HTTP/1.1
Host: resource-controller.cloud.ibm.com
User-Agent: platform-services-go-sdk/0.22.3 (lang=go; arch=amd64; os=linux; go.version=go1.16.10)
Authorization: [redacted]
X-Original-User-Agent: terraform-provider-ibm/1.36.0
Accept-Encoding: gzip

: timestamp=2022-01-26T06:28:57.678Z
2022-01-26T06:28:57.679Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:57 [DEBUG] DELETE https://resource-controller.cloud.ibm.com/v2/resource_groups/4b1a9955550b4a59b1e9db20a39f7edf: timestamp=2022-01-26T06:28:57.678Z
2022-01-26T06:28:58.280Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:58 [Debug] Response:
HTTP/2.0 204 No Content
Connection: close
Cache-Control: max-age=0, no-cache, no-store
Date: Wed, 26 Jan 2022 06:28:58 GMT
Expires: Wed, 26 Jan 2022 06:28:58 GMT
Pragma: no-cache
Server: istio-envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: 85093032433
X-Envoy-Upstream-Service-Time: 468
X-Request-Id: 85093032433
X-Response-Time: 457.720ms
_request_id: 85093032433

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

provider registry.terraform.io/ibm-cloud/ibm v1.36.0

Affected Resource(s)

• ibm_resource_group

Terraform Configuration Files

internal url: https://github.ibm.com/GoldenEye/iam-user-apikey-secrets-manager-module/tree/master/examples/complete

Debug Output

iam-user-apikey-secrets-manager-module-test-logs-220126-062858.tar.gz

Panic Output

Expected Behavior

Destroy should pass after successfully deleting the resource group

Actual Behavior

204 response from RC

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[kavya498]

@ocofaigh ,
After looking into the logs, it is not the issue with resource group deletion.. It is issue with service id, looks like service id has apikey and it is causing the error..

DELETE /v1/serviceids/ServiceId-8fbb7cc1-dcb4-4b07-bbec-b98bde1cd169 HTTP/1.1
Host: iam.cloud.ibm.com
User-Agent: platform-services-go-sdk/0.22.3 (lang=go; arch=amd64; os=linux; go.version=go1.16.10)
Authorization: [redacted]
X-Original-User-Agent: terraform-provider-ibm/1.36.0
Accept-Encoding: gzip

: timestamp=2022-01-26T06:28:42.306Z
2022-01-26T06:28:42.307Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:42 [DEBUG] DELETE https://iam.cloud.ibm.com/v1/serviceids/ServiceId-8fbb7cc1-dcb4-4b07-bbec-b98bde1cd169: timestamp=2022-01-26T06:28:42.306Z
2022-01-26T06:28:42.320Z [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 30
2022-01-26T06:28:42.320Z [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate.d/sm-iam-user-apikey-zuyblp/terraform.tfstate
2022-01-26T06:28:42.324Z [TRACE] vertex "module.secrets_manager_group_service.time_sleep.wait_30_seconds (destroy)": visit complete
2022-01-26T06:28:42.325Z [TRACE] vertex "module.secrets_manager_group_service.restapi_object.secret_group (destroy)": starting visit (*terraform.NodeDestroyResourceInstance)
2022-01-26T06:28:42.325Z [TRACE] readDiff: Read Delete change from plan for module.secrets_manager_group_service.restapi_object.secret_group
2022-01-26T06:28:42.325Z [TRACE] readResourceInstanceState: reading state for module.secrets_manager_group_service.restapi_object.secret_group
2022-01-26T06:28:42.326Z [TRACE] upgradeResourceState: schema version of module.secrets_manager_group_service.restapi_object.secret_group is still 0; calling provider "restapi" for any other minor fixups
2022-01-26T06:28:42.326Z [TRACE] GRPCProvider: UpgradeResourceState
2022-01-26T06:28:42.328Z [INFO]  Starting apply for module.secrets_manager_group_service.restapi_object.secret_group
2022-01-26T06:28:42.329Z [DEBUG] module.secrets_manager_group_service.restapi_object.secret_group: applying the planned Delete change
2022-01-26T06:28:42.329Z [TRACE] GRPCProvider: ApplyResourceChange
2022-01-26T06:28:42.331Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 2022/01/26 06:28:42 common.go: make_api_object routine called for id '36dfb8d7-5363-1ab2-bb34-8abcfcdfe04c'
2022-01-26T06:28:42.331Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 2022/01/26 06:28:42 create_path: 
2022-01-26T06:28:42.331Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 2022/01/26 06:28:42 resource_api_object.go: Delete routine called. Object built:
2022-01-26T06:28:42.332Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: id: 36dfb8d7-5363-1ab2-bb34-8abcfcdfe04c
2022-01-26T06:28:42.332Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: get_path: //a542cff0-6efb-460b-910b-b12cff7f6d58.jp-tok.secrets-manager.appdomain.cloud/api/v1/secret_groups/{id}
2022-01-26T06:28:42.332Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: post_path: //a542cff0-6efb-460b-910b-b12cff7f6d58.jp-tok.secrets-manager.appdomain.cloud/api/v1/secret_groups
2022-01-26T06:28:42.332Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: put_path: //a542cff0-6efb-460b-910b-b12cff7f6d58.jp-tok.secrets-manager.appdomain.cloud/api/v1/secret_groups/{id}
2022-01-26T06:28:42.332Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: delete_path: //a542cff0-6efb-460b-910b-b12cff7f6d58.jp-tok.secrets-manager.appdomain.cloud/api/v1/secret_groups/{id}
2022-01-26T06:28:42.333Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: create_method: POST
2022-01-26T06:28:42.333Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: read_method: GET
2022-01-26T06:28:42.333Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: update_method: PUT
2022-01-26T06:28:42.333Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: destroy_method: DELETE
2022-01-26T06:28:42.334Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: debug: false
2022-01-26T06:28:42.334Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: read_search: (map[string]string) {
2022-01-26T06:28:42.334Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: }
2022-01-26T06:28:42.334Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 
2022-01-26T06:28:42.334Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: data: (map[string]interface {}) (len=2) {
2022-01-26T06:28:42.335Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:  (string) (len=8) "metadata": (map[string]interface {}) (len=2) {
2022-01-26T06:28:42.335Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:   (string) (len=15) "collection_type": (string) (len=53) "application/vnd.ibm.secrets-manager.secret.group+json",
2022-01-26T06:28:42.335Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:   (string) (len=16) "collection_total": (float64) 1
2022-01-26T06:28:42.335Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:  },
2022-01-26T06:28:42.336Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:  (string) (len=9) "resources": ([]interface {}) (len=1 cap=1) {
2022-01-26T06:28:42.336Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:   (map[string]interface {}) (len=2) {
2022-01-26T06:28:42.336Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:    (string) (len=4) "name": (string) (len=42) "sm-iam-user-apikey-zuyblp-svc-secret-group",
2022-01-26T06:28:42.336Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:    (string) (len=11) "description": (string) (len=20) "service secret group"
2022-01-26T06:28:42.336Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:   }
2022-01-26T06:28:42.337Z [DEBUG] provider.terraform-provider-restapi_v1.15.1:  }
2022-01-26T06:28:42.337Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: }
2022-01-26T06:28:42.337Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 
2022-01-26T06:28:42.337Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: api_data: (map[string]interface {}) {
2022-01-26T06:28:42.337Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: }
2022-01-26T06:28:42.338Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 
2022-01-26T06:28:42.338Z [DEBUG] provider.terraform-provider-restapi_v1.15.1: 
2022-01-26T06:28:42.533Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:42 [Debug] Response:
HTTP/1.1 409 Conflict
Content-Length: 334
Akamai-Grn: 0.753b2f17.1643178522.5fe21406
Cache-Control: no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Language: en-US
Content-Type: application/json
Date: Wed, 26 Jan 2022 06:28:42 GMT
Expires: 0
Pragma: no-cache
Set-Cookie: sessioncookie="82566fefa9ea493d"; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: aWFtaWQtNy4xLTEyNjY2LWUwY2Q2ZGUtNjZiZGI5YmNiOC1namxudA-d96f73568e214024b64bd97dcbd33607
X-Content-Type-Options: nosniff
X-Proxy-Upstream-Service-Time: 160

{"errors":[{"code":"delete_not_possible","message_code":"BXNIM0105E","message":"Unable to delete object","details":"Object type 'ApiKey' with ID 'ServiceId-8fbb7cc1-dcb4-4b07-bbec-b98bde1cd169' could not be deleted"}],"trace":"aWFtaWQtNy4xLTEyNjY2LWUwY2Q2ZGUtNjZiZGI5YmNiOC1namxudA-d96f73568e214024b64bd97dcbd33607","status_code":409}: timestamp=2022-01-26T06:28:42.532Z
2022-01-26T06:28:42.534Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2022/01/26 06:28:42 Error deleting serviceID: Unable to delete object {
    "StatusCode": 409,
    "Headers": {
        "Akamai-Grn": [
            "0.753b2f17.1643178522.5fe21406"
        ],
        "Cache-Control": [
            "no-cache, no-store, must-revalidate"
        ],
        "Connection": [
            "keep-alive"
        ],
        "Content-Language": [
            "en-US"
        ],
        "Content-Length": [
            "334"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Wed, 26 Jan 2022 06:28:42 GMT"
        ],
        "Expires": [
            "0"
        ],
        "Pragma": [
            "no-cache"
        ],
        "Set-Cookie": [
            "sessioncookie=\"82566fefa9ea493d\"; Path=/; Secure; HttpOnly"
        ],
        "Strict-Transport-Security": [
            "max-age=31536000; includeSubDomains"
        ],
        "Transaction-Id": [
            "aWFtaWQtNy4xLTEyNjY2LWUwY2Q2ZGUtNjZiZGI5YmNiOC1namxudA-d96f73568e214024b64bd97dcbd33607"
        ],
        "X-Content-Type-Options": [
            "nosniff"
        ],
        "X-Proxy-Upstream-Service-Time": [
            "160"
        ]
    },
    "Result": {
        "errors": [
            {
                "code": "delete_not_possible",
                "details": "Object type 'ApiKey' with ID 'ServiceId-8fbb7cc1-dcb4-4b07-bbec-b98bde1cd169' could not be deleted",
                "message": "Unable to delete object",
                "message_code": "BXNIM0105E"
            }
        ],
        "status_code": 409,
        "trace": "aWFtaWQtNy4xLTEyNjY2LWUwY2Q2ZGUtNjZiZGI5YmNiOC1namxudA-d96f73568e214024b64bd97dcbd33607"
    },
    "RawResult": null
}: timestamp=2022-01-26T06:28:42.533Z

[ocofaigh]

@kavya498 Thank you for looking into this. It was very hard to actually know what the failure was here. Could the details in the json error be output as part of the standard output error? Simply printing Error: Unable to delete object was not very useful :)

[kavya498]

@ocofaigh , It is the error message from API,
May be we can add some kind of prefix like Error deleting service id in provider, inorder to identify which resource has thrown this..

[ocofaigh]

yes I think that would help very much @kavya498