aws_db_instance performance_insights_enabled HTTP 500 error

[jrobison-sb]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.11.11
+ provider.archive v1.0.3
+ provider.aws v2.12.0
+ provider.template v1.0.0

Affected Resource(s)

• aws_db_instance (resource)

Terraform Configuration Files

performance_insights_enabled = "true"
performance_insights_retention_period = 7

Debug Output

2019-05-28T10:57:37.946-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 2019/05/28 10:57:37 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request rds/ModifyDBInstance, attempt 13
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 2019/05/28 10:57:37 [DEBUG] [aws-sdk-go] DEBUG: Request rds/ModifyDBInstance Details:
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: POST / HTTP/1.1
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Host: rds.us-east-1.amazonaws.com
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: User-Agent: aws-sdk-go/1.19.36 (go1.12.5; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.0
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Content-Length: 227
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20190528/us-east-1/rds/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=REDACTED
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: X-Amz-Date: 20190528T145737Z
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Accept-Encoding: gzip
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Action=ModifyDBInstance&ApplyImmediately=true&DBInstanceIdentifier=staging-mydb-20181106191503232000000001&EnablePerformanceInsights=true&PerformanceInsightsKMSKeyId=&PerformanceInsightsRetentionPeriod=7&Version=2014-10-31
2019-05-28T10:57:37.947-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: -----------------------------------------------------
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 2019/05/28 10:57:38 [DEBUG] [aws-sdk-go] DEBUG: Response rds/ModifyDBInstance Details:
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: ---[ RESPONSE ]--------------------------------------
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: HTTP/1.1 500 Internal Server Error
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Connection: close
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Content-Length: 224
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Content-Type: text/xml
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: Date: Tue, 28 May 2019 14:57:37 GMT
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: X-Amzn-Requestid: e405aaa9-e7e1-4daa-bd54-f5299a93d667
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: -----------------------------------------------------
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 2019/05/28 10:57:38 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4:   <Error>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4:     <Type>Receiver</Type>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4:     <Code>InternalFailure</Code>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4:   </Error>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4:   <RequestId>e405aaa9-e7e1-4daa-bd54-f5299a93d667</RequestId>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: </ErrorResponse>
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 2019/05/28 10:57:38 [DEBUG] [aws-sdk-go] DEBUG: Validate Response rds/ModifyDBInstance failed, not retrying, error InternalFailure:
2019-05-28T10:57:38.221-0400 [DEBUG] plugin.terraform-provider-aws_v2.12.0_x4: 	status code: 500, request id: e405aaa9-e7e1-4daa-bd54-f5299a93d667

Panic Output

No panic.

Expected Behavior

Terraform should successfully update/modify my aws_db_instance to turn on Performance Insights.

Actual Behavior

It keeps retrying and 500'ing forever until eventually timing out.

Steps to Reproduce

1. Create an aws_db_instance, without enabling PI.
2. Set performance_insights_enabled = "true"
3. terraform apply and watch it enter a retry-loop that will never work, and then eventually timeout with an HTTP 500 error.

Important Factoids

FWIW, here's the perspective of Cloudtrail:

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "REDACTED",
        "arn": "arn:aws:iam::1234567890:user/me",
        "accountId": "1234567890",
        "accessKeyId": "REDACTED",
        "userName": "me"
    },
    "eventTime": "2019-05-28T13:48:34Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "ModifyDBInstance",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "1.2.3.4",
    "userAgent": "aws-sdk-go/1.19.36 (go1.12.5; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.0",
    "errorCode": "InternalFailure",
    "errorMessage": "An unknown error occurred",
    "requestParameters": {
        "allowMajorVersionUpgrade": false,
        "performanceInsightsKMSKeyId": "",
        "enablePerformanceInsights": true,
        "applyImmediately": true,
        "dBInstanceIdentifier": "staging-mydb-20181106191503232000000001",
        "performanceInsightsRetentionPeriod": 7
    },
    "responseElements": null,
    "requestID": "e6a80667-0175-4adf-9c72-5d60215bab0d",
    "eventID": "be9e0fd5-bdfe-46e6-8386-ab263b4d5d31",
    "eventType": "AwsApiCall",
    "recipientAccountId": "1234567890"
}

References

• Performance Insights configuration for aws_db_instance #6453

[bflad]

Hi @jrobison-sb 👋 Sorry that this new feature in the Terraform resource is giving you trouble.

Given this 500 InternalFailure error is being returned from the RDS API, have you tried reaching out to AWS Support? The Terraform AWS Provider maintainers will not have insight into why the API is responding in this manner.

You may want to also verify that your RDS database instance meets the version requirements for Performance Insights, which can be found here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html

Please do reach out if there is anything specific in the Terraform resource that AWS Support notes as problematic.

[jrobison-sb]

@bflad Thanks for your reply. I opened a ticket with AWS support and they found the cause of the error in their logs: The specified KMS key [] does not exist, is not enabled or you do not have permissions to access it. The two lines of code I pasted in my initial post are the only PI-related lines I have, I'm not specifying any KMS key because that parameter is supposed to be optional.

The RDS API docs say that specifying a KMS key is not required, and if no key is chosen then the default KMS key will be used.

After a little more digging, AWS support able to reproduce the same issue using the awscli, which I'll paste below:

Now, I went ahead and tested this using AWS CLI and I was able to reproduce the same error.

First I did not specify KMS key ID:

egidij@uf694b65c34535c:~$ aws rds modify-db-instance --db-instance-identifier aaaaa --enable-performance-insights

API Call:

{

  "applyImmediately": false,
  "enablePerformanceInsights": true,
  "performanceInsightsKMSKeyId": null,
  "performanceInsightsRetentionPeriod": null,
  "dbinstanceIdentifier": "aaaaa"
}

As expected it worked as it should do, default KMS key was used and no errors.

Now I created another instance and ran the following command using AWS CLI, please note, here I specified an empty string for PerformanceInsightsKMSKeyId and I was able to get the same error on the API call internally.

egidij@uf694b65c34535c:~$ aws rds modify-db-instance --db-instance-identifier qqqqq --enable-performance-insights --performance-insights-kms-key-id "" 

An error occurred (InternalFailure) when calling the ModifyDBInstance operation (reached max retries: 4): Unknown

API Call:

{
  "applyImmediately": false,
  "enablePerformanceInsights": true,
  "performanceInsightsKMSKeyId": "",
  "performanceInsightsRetentionPeriod": null,
  "dbinstanceIdentifier": "qqqqq"
}

With the following API error - "The specified KMS key [] does not exist, is not enabled or you do not have permissions to access it."

As you can see here I passed in "" to --performance-insights-kms-key-id and it failed with same error internally, same was noticed by me when checking your API call.

So this leads me to believe the issue is originating from terraform, it looks like an empty string is being passed in when noting at all should be passed.

Does this help to clarify the problem? And do you agree that Terraform is sending an empty string for the KMS key when it shouldn't be passing performanceInsightsKMSKeyId at all?

Thanks again.

[jrobison-sb]

Any additional info needed here?

[marpada]

I bumped into the same problem. Interestingly enough the console doesn't allow an empty kms key, and uses aws/rds by default, so as a workaround I added the following to my terraform config:

data "aws_kms_key" "aws_rds" {
  key_id = "alias/aws/rds"
  }
...

  performance_insights_enabled = true
  performance_insights_kms_key_id  = "${data.aws_kms_key.aws_rds.arn}"
}

[bflad]

Fix submitted: #9745

[bflad]

The fix for this has been merged and will release with version 2.24.0 of the Terraform AWS Provider, tomorrow. 👍

[ghost]

This has been released in version 2.24.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!