Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to metric_query on aws_cloudwatch_metric_alarm fail if dimensions is set #7593

Closed
strongoose opened this issue Feb 18, 2019 · 8 comments · Fixed by #8085
Closed

Updates to metric_query on aws_cloudwatch_metric_alarm fail if dimensions is set #7593

strongoose opened this issue Feb 18, 2019 · 8 comments · Fixed by #8085
Labels
bug Addresses a defect in current functionality. service/cloudwatch Issues and PRs that pertain to the cloudwatch service.
Milestone

Comments

@strongoose
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

+ provider.aws v1.59.0```

### Affected Resource(s)

<!--- Please list the affected resources and data sources. --->
* aws_cloudwatch_metric_alarm

### Terraform Configuration Files

<!--- Information about code formatting: https://help.github.com/articles/basic-writing-and-formatting-syntax/#quoting-code --->

```hcl
resource aws_cloudwatch_metric_alarm test {
  alarm_name                = "TEST"
  alarm_description         = "Debugging metric update in terraform"
  comparison_operator       = "GreaterThanOrEqualToThreshold"
  evaluation_periods        = "1"
  threshold                 = 10
  insufficient_data_actions = []
  ok_actions                = []
  alarm_actions             = []

  metric_query = {
    id          = "e2"
    expression  = "m1*100/m2"
    label       = "5XX percentage"
    return_data = "true"
  }

  metric_query {
    id = "m1"

    metric {
      metric_name = "5XXError"
      namespace   = "AWS/ApiGateway"
      period      = "90"
      stat        = "Sum"
      unit        = "Count"

      dimensions {
        ApiName = "Test"
        Stage   = "Test"
      }
    }
  }

  metric_query {
    id = "m2"

    metric {
      metric_name = "Count"
      namespace   = "AWS/ApiGateway"
      period      = "90"
      stat        = "Sum"
      unit        = "Count"

      dimensions {
        ApiName = "Test"
        Stage   = "Test"
      }
    }
  }
}

To reproduce the error, make any change to the dimension metric queries, for example change metric_name on m1 to 4XXError.

Expected Behavior

The metric query is updates as expected to alert on percentage of 4XXs

Actual Behavior

Apply fails with:


1 error(s) occurred:

* aws_cloudwatch_metric_alarm.test: 1 error(s) occurred:

* aws_cloudwatch_metric_alarm.test: Updating metric alarm failed: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutMetricAlarmInput.Metrics[3].Id.


Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Steps to Reproduce

  1. terraform apply the HCL above
  2. change one of the metrics, e.g. change metric_name on line 22 to 4XXError
  3. run terraform apply again

Important Factoids

This only seems to affect metrics with dimensions.

@strongoose
Copy link
Contributor Author

strongoose commented Feb 18, 2019

Having looked at the state, it seems that the update operation creates a 4th empty metric with no ID:

"metric_query.3256485992.expression": "",
"metric_query.3256485992.id": "",
"metric_query.3256485992.label": "",
"metric_query.3256485992.metric.#": "0",
"metric_query.3256485992.return_data": "false",

this leaves terraform in an unrecoverable state until the resource is destroyed

@sjauld
Copy link
Contributor

sjauld commented Mar 21, 2019

I can confirm that the address of PutMetricAlarmInput.Metrics[x].Id. matches the metric query that is being blanked by terraform

@luthor2016ad
Copy link

luthor2016ad commented Mar 25, 2019

I confirm the issue
the only way to update the alarm is to destoy it before...

sjauld added a commit to sjauld/terraform-provider-aws that referenced this issue Mar 26, 2019
-- Test output below --

aws-vault exec sand-superuser -- make testacc TESTARGS="-run TestAccAWSCloudWatchMetricAlarm_expression"
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -parallel 20 -run TestAccAWSCloudWatchMetricAlarm_expression -timeout 120m
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSCloudWatchMetricAlarm_expression
=== PAUSE TestAccAWSCloudWatchMetricAlarm_expression
=== CONT  TestAccAWSCloudWatchMetricAlarm_expression
2019/03/27 09:00:55 [WARN] Test: Step plan: DIFF:

CREATE: aws_cloudwatch_metric_alarm.foobar
  actions_enabled:                                        "" => "true"
  alarm_description:                                      "" => "This metric monitors ec2 cpu utilization"
  alarm_name:                                             "" => "terraform-test-foobar6453812907472873677" (forces new resource)
  arn:                                                    "" => "<computed>"
  comparison_operator:                                    "" => "GreaterThanOrEqualToThreshold"
  evaluate_low_sample_count_percentiles:                  "" => "<computed>"
  evaluation_periods:                                     "" => "2"
  metric_query.#:                                         "" => "1"
  metric_query.2835246568.expression:                     "" => "m1"
  metric_query.2835246568.id:                             "" => "e1"
  metric_query.2835246568.label:                          "" => "cat"
  metric_query.2835246568.metric.#:                       "" => "1"
  metric_query.2835246568.metric.0.dimensions.%:          "" => "1"
  metric_query.2835246568.metric.0.dimensions.InstanceId: "" => "i-abc123"
  metric_query.2835246568.metric.0.metric_name:           "" => "CPUUtilization"
  metric_query.2835246568.metric.0.namespace:             "" => "AWS/EC2"
  metric_query.2835246568.metric.0.period:                "" => "120"
  metric_query.2835246568.metric.0.stat:                  "" => "Average"
  metric_query.2835246568.metric.0.unit:                  "" => "Count"
  metric_query.2835246568.return_data:                    "" => "false"
  threshold:                                              "" => "80"
  treat_missing_data:                                     "" => "missing"

STATE:

<no state>
2019/03/27 09:00:58 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error occurred:
        * aws_cloudwatch_metric_alarm.foobar: No metric_query may have both `expression` and a `metric` specified

2019/03/27 09:00:58 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error occurred:
        * aws_cloudwatch_metric_alarm.foobar: No metric_query may have both `expression` and a `metric` specified

2019/03/27 09:01:02 [WARN] Test: Step plan: DIFF:

CREATE: aws_cloudwatch_metric_alarm.foobar
  actions_enabled:                                        "" => "true"
  alarm_description:                                      "" => "This metric monitors ec2 cpu utilization"
  alarm_name:                                             "" => "terraform-test-foobar6453812907472873677" (forces new resource)
  arn:                                                    "" => "<computed>"
  comparison_operator:                                    "" => "GreaterThanOrEqualToThreshold"
  evaluate_low_sample_count_percentiles:                  "" => "<computed>"
  evaluation_periods:                                     "" => "2"
  metric_query.#:                                         "" => "2"
  metric_query.2520032513.expression:                     "" => ""
  metric_query.2520032513.id:                             "" => "m1"
  metric_query.2520032513.label:                          "" => ""
  metric_query.2520032513.metric.#:                       "" => "1"
  metric_query.2520032513.metric.0.dimensions.%:          "" => "1"
  metric_query.2520032513.metric.0.dimensions.InstanceId: "" => "i-abc123"
  metric_query.2520032513.metric.0.metric_name:           "" => "CPUUtilization"
  metric_query.2520032513.metric.0.namespace:             "" => "AWS/EC2"
  metric_query.2520032513.metric.0.period:                "" => "120"
  metric_query.2520032513.metric.0.stat:                  "" => "Average"
  metric_query.2520032513.metric.0.unit:                  "" => "Count"
  metric_query.2520032513.return_data:                    "" => "false"
  metric_query.3055748110.expression:                     "" => "m1"
  metric_query.3055748110.id:                             "" => "e1"
  metric_query.3055748110.label:                          "" => "cat"
  metric_query.3055748110.metric.#:                       "" => "0"
  metric_query.3055748110.return_data:                    "" => "true"
  threshold:                                              "" => "80"
  treat_missing_data:                                     "" => "missing"

STATE:

<no state>
2019/03/27 09:01:26 [WARN] Test: Step plan: DIFF:

UPDATE: aws_cloudwatch_metric_alarm.foobar
  metric_query.#:                                         "2" => "3"
  metric_query.2520032513.expression:                     "" => ""
  metric_query.2520032513.id:                             "m1" => "m1"
  metric_query.2520032513.label:                          "" => ""
  metric_query.2520032513.metric.#:                       "1" => "1"
  metric_query.2520032513.metric.0.dimensions.InstanceId: "i-abc123" => "i-abc123"
  metric_query.2520032513.metric.0.metric_name:           "CPUUtilization" => "CPUUtilization"
  metric_query.2520032513.metric.0.namespace:             "AWS/EC2" => "AWS/EC2"
  metric_query.2520032513.metric.0.period:                "120" => "120"
  metric_query.2520032513.metric.0.stat:                  "Average" => "Average"
  metric_query.2520032513.metric.0.unit:                  "Count" => "Count"
  metric_query.2520032513.return_data:                    "false" => "false"
  metric_query.2939696463.expression:                     "" => "m1"
  metric_query.2939696463.id:                             "" => "e1"
  metric_query.2939696463.label:                          "" => "cat"
  metric_query.2939696463.metric.#:                       "0" => "0"
  metric_query.2939696463.return_data:                    "" => "false"
  metric_query.3055748110.expression:                     "m1" => ""
  metric_query.3055748110.id:                             "e1" => ""
  metric_query.3055748110.label:                          "cat" => ""
  metric_query.3055748110.metric.#:                       "0" => "0"
  metric_query.3055748110.return_data:                    "true" => "false"
  metric_query.3779742054.expression:                     "" => "e1"
  metric_query.3779742054.id:                             "" => "e2"
  metric_query.3779742054.label:                          "" => "bug"
  metric_query.3779742054.metric.#:                       "0" => "0"
  metric_query.3779742054.return_data:                    "" => "true"

STATE:

aws_cloudwatch_metric_alarm.foobar:
  ID = terraform-test-foobar6453812907472873677
  provider = provider.aws
  actions_enabled = true
  alarm_actions.# = 0
  alarm_description = This metric monitors ec2 cpu utilization
  alarm_name = terraform-test-foobar6453812907472873677
  arn = arn:aws:cloudwatch:us-west-2:130475909466:alarm:terraform-test-foobar6453812907472873677
  comparison_operator = GreaterThanOrEqualToThreshold
  datapoints_to_alarm = 0
  dimensions.% = 0
  evaluate_low_sample_count_percentiles =
  evaluation_periods = 2
  extended_statistic =
  insufficient_data_actions.# = 0
  metric_name =
  metric_query.# = 2
  metric_query.2520032513.expression =
  metric_query.2520032513.id = m1
  metric_query.2520032513.label =
  metric_query.2520032513.metric.# = 1
  metric_query.2520032513.metric.0.dimensions.% = 1
  metric_query.2520032513.metric.0.dimensions.InstanceId = i-abc123
  metric_query.2520032513.metric.0.metric_name = CPUUtilization
  metric_query.2520032513.metric.0.namespace = AWS/EC2
  metric_query.2520032513.metric.0.period = 120
  metric_query.2520032513.metric.0.stat = Average
  metric_query.2520032513.metric.0.unit = Count
  metric_query.2520032513.return_data = false
  metric_query.3055748110.expression = m1
  metric_query.3055748110.id = e1
  metric_query.3055748110.label = cat
  metric_query.3055748110.metric.# = 0
  metric_query.3055748110.return_data = true
  namespace =
  ok_actions.# = 0
  period = 0
  statistic =
  threshold = 80
  treat_missing_data = missing
  unit =
2019/03/27 09:01:50 [WARN] Test: Step plan: DIFF:

UPDATE: aws_cloudwatch_metric_alarm.foobar
  metric_query.#:                                         "3" => "2"
  metric_query.2520032513.expression:                     "" => ""
  metric_query.2520032513.id:                             "m1" => "m1"
  metric_query.2520032513.label:                          "" => ""
  metric_query.2520032513.metric.#:                       "1" => "1"
  metric_query.2520032513.metric.0.dimensions.InstanceId: "i-abc123" => "i-abc123"
  metric_query.2520032513.metric.0.metric_name:           "CPUUtilization" => "CPUUtilization"
  metric_query.2520032513.metric.0.namespace:             "AWS/EC2" => "AWS/EC2"
  metric_query.2520032513.metric.0.period:                "120" => "120"
  metric_query.2520032513.metric.0.stat:                  "Average" => "Average"
  metric_query.2520032513.metric.0.unit:                  "Count" => "Count"
  metric_query.2520032513.return_data:                    "false" => "false"
  metric_query.2939696463.expression:                     "m1" => ""
  metric_query.2939696463.id:                             "e1" => ""
  metric_query.2939696463.label:                          "cat" => ""
  metric_query.2939696463.metric.#:                       "0" => "0"
  metric_query.2939696463.return_data:                    "false" => "false"
  metric_query.3055748110.expression:                     "" => "m1"
  metric_query.3055748110.id:                             "" => "e1"
  metric_query.3055748110.label:                          "" => "cat"
  metric_query.3055748110.metric.#:                       "0" => "0"
  metric_query.3055748110.return_data:                    "" => "true"
  metric_query.3779742054.expression:                     "e1" => ""
  metric_query.3779742054.id:                             "e2" => ""
  metric_query.3779742054.label:                          "bug" => ""
  metric_query.3779742054.metric.#:                       "0" => "0"
  metric_query.3779742054.return_data:                    "true" => "false"

STATE:

aws_cloudwatch_metric_alarm.foobar:
  ID = terraform-test-foobar6453812907472873677
  provider = provider.aws
  actions_enabled = true
  alarm_actions.# = 0
  alarm_description = This metric monitors ec2 cpu utilization
  alarm_name = terraform-test-foobar6453812907472873677
  arn = arn:aws:cloudwatch:us-west-2:130475909466:alarm:terraform-test-foobar6453812907472873677
  comparison_operator = GreaterThanOrEqualToThreshold
  datapoints_to_alarm = 0
  dimensions.% = 0
  evaluate_low_sample_count_percentiles =
  evaluation_periods = 2
  extended_statistic =
  insufficient_data_actions.# = 0
  metric_name =
  metric_query.# = 3
  metric_query.2520032513.expression =
  metric_query.2520032513.id = m1
  metric_query.2520032513.label =
  metric_query.2520032513.metric.# = 1
  metric_query.2520032513.metric.0.dimensions.% = 1
  metric_query.2520032513.metric.0.dimensions.InstanceId = i-abc123
  metric_query.2520032513.metric.0.metric_name = CPUUtilization
  metric_query.2520032513.metric.0.namespace = AWS/EC2
  metric_query.2520032513.metric.0.period = 120
  metric_query.2520032513.metric.0.stat = Average
  metric_query.2520032513.metric.0.unit = Count
  metric_query.2520032513.return_data = false
  metric_query.2939696463.expression = m1
  metric_query.2939696463.id = e1
  metric_query.2939696463.label = cat
  metric_query.2939696463.metric.# = 0
  metric_query.2939696463.return_data = false
  metric_query.3779742054.expression = e1
  metric_query.3779742054.id = e2
  metric_query.3779742054.label = bug
  metric_query.3779742054.metric.# = 0
  metric_query.3779742054.return_data = true
  namespace =
  ok_actions.# = 0
  period = 0
  statistic =
  threshold = 80
  treat_missing_data = missing
  unit =
2019/03/27 09:02:16 [WARN] Test: Step plan: DIFF:

UPDATE: aws_cloudwatch_metric_alarm.foobar
  metric_query.156515512.expression:                      "" => ""
  metric_query.156515512.id:                              "" => "m1"
  metric_query.156515512.label:                           "" => ""
  metric_query.156515512.metric.#:                        "0" => "1"
  metric_query.156515512.metric.0.dimensions.%:           "0" => "1"
  metric_query.156515512.metric.0.dimensions.InstanceId:  "" => "i-abc123"
  metric_query.156515512.metric.0.metric_name:            "" => "CPUUtilization"
  metric_query.156515512.metric.0.namespace:              "" => "AWS/EC2"
  metric_query.156515512.metric.0.period:                 "" => "120"
  metric_query.156515512.metric.0.stat:                   "" => "Maximum"
  metric_query.156515512.metric.0.unit:                   "" => "Count"
  metric_query.156515512.return_data:                     "" => "false"
  metric_query.2520032513.expression:                     "" => ""
  metric_query.2520032513.id:                             "m1" => ""
  metric_query.2520032513.label:                          "" => ""
  metric_query.2520032513.metric.#:                       "1" => "0"
  metric_query.2520032513.metric.0.dimensions.%:          "1" => "0"
  metric_query.2520032513.metric.0.dimensions.InstanceId: "i-abc123" => ""
  metric_query.2520032513.metric.0.metric_name:           "CPUUtilization" => ""
  metric_query.2520032513.metric.0.namespace:             "AWS/EC2" => ""
  metric_query.2520032513.metric.0.period:                "120" => "0"
  metric_query.2520032513.metric.0.stat:                  "Average" => ""
  metric_query.2520032513.metric.0.unit:                  "Count" => ""
  metric_query.2520032513.return_data:                    "false" => "false"
  metric_query.3055748110.expression:                     "m1" => "m1"
  metric_query.3055748110.id:                             "e1" => "e1"
  metric_query.3055748110.label:                          "cat" => "cat"
  metric_query.3055748110.metric.#:                       "0" => "0"
  metric_query.3055748110.return_data:                    "true" => "true"

STATE:

aws_cloudwatch_metric_alarm.foobar:
  ID = terraform-test-foobar6453812907472873677
  provider = provider.aws
  actions_enabled = true
  alarm_actions.# = 0
  alarm_description = This metric monitors ec2 cpu utilization
  alarm_name = terraform-test-foobar6453812907472873677
  arn = arn:aws:cloudwatch:us-west-2:130475909466:alarm:terraform-test-foobar6453812907472873677
  comparison_operator = GreaterThanOrEqualToThreshold
  datapoints_to_alarm = 0
  dimensions.% = 0
  evaluate_low_sample_count_percentiles =
  evaluation_periods = 2
  extended_statistic =
  insufficient_data_actions.# = 0
  metric_name =
  metric_query.# = 2
  metric_query.2520032513.expression =
  metric_query.2520032513.id = m1
  metric_query.2520032513.label =
  metric_query.2520032513.metric.# = 1
  metric_query.2520032513.metric.0.dimensions.% = 1
  metric_query.2520032513.metric.0.dimensions.InstanceId = i-abc123
  metric_query.2520032513.metric.0.metric_name = CPUUtilization
  metric_query.2520032513.metric.0.namespace = AWS/EC2
  metric_query.2520032513.metric.0.period = 120
  metric_query.2520032513.metric.0.stat = Average
  metric_query.2520032513.metric.0.unit = Count
  metric_query.2520032513.return_data = false
  metric_query.3055748110.expression = m1
  metric_query.3055748110.id = e1
  metric_query.3055748110.label = cat
  metric_query.3055748110.metric.# = 0
  metric_query.3055748110.return_data = true
  namespace =
  ok_actions.# = 0
  period = 0
  statistic =
  threshold = 80
  treat_missing_data = missing
  unit =
2019/03/27 09:02:19 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error occurred:
        * aws_cloudwatch_metric_alarm.foobar: Updating metric alarm failed: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutMetricAlarmInput.Metrics[2].Id.

2019/03/27 09:02:19 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error occurred:
        * aws_cloudwatch_metric_alarm.foobar: Updating metric alarm failed: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutMetricAlarmInput.Metrics[2].Id.

2019/03/27 09:02:19 [WARN] Test: Executing destroy step
2019/03/27 09:02:24 [WARN] Test: Step plan: DIFF:

DESTROY: aws_cloudwatch_metric_alarm.foobar

STATE:

aws_cloudwatch_metric_alarm.foobar:
  ID = terraform-test-foobar6453812907472873677
  provider = provider.aws
  actions_enabled = true
  alarm_actions.# = 0
  alarm_description = This metric monitors ec2 cpu utilization
  alarm_name = terraform-test-foobar6453812907472873677
  arn = arn:aws:cloudwatch:us-west-2:130475909466:alarm:terraform-test-foobar6453812907472873677
  comparison_operator = GreaterThanOrEqualToThreshold
  datapoints_to_alarm = 0
  dimensions.% = 0
  evaluate_low_sample_count_percentiles =
  evaluation_periods = 2
  extended_statistic =
  insufficient_data_actions.# = 0
  metric_name =
  metric_query.# = 2
  metric_query.2520032513.expression =
  metric_query.2520032513.id = m1
  metric_query.2520032513.label =
  metric_query.2520032513.metric.# = 1
  metric_query.2520032513.metric.0.dimensions.% = 1
  metric_query.2520032513.metric.0.dimensions.InstanceId = i-abc123
  metric_query.2520032513.metric.0.metric_name = CPUUtilization
  metric_query.2520032513.metric.0.namespace = AWS/EC2
  metric_query.2520032513.metric.0.period = 120
  metric_query.2520032513.metric.0.stat = Average
  metric_query.2520032513.metric.0.unit = Count
  metric_query.2520032513.return_data = false
  metric_query.3055748110.expression = m1
  metric_query.3055748110.id = e1
  metric_query.3055748110.label = cat
  metric_query.3055748110.metric.# = 0
  metric_query.3055748110.return_data = true
  namespace =
  ok_actions.# = 0
  period = 0
  statistic =
  threshold = 80
  treat_missing_data = missing
  unit =
--- FAIL: TestAccAWSCloudWatchMetricAlarm_expression (104.76s)
    testing.go:538: Step 4 error: Error applying: 1 error occurred:
                * aws_cloudwatch_metric_alarm.foobar: 1 error occurred:
                * aws_cloudwatch_metric_alarm.foobar: Updating metric alarm failed: InvalidParameter: 1 validation error(s) found.
        - minimum field size of 1, PutMetricAlarmInput.Metrics[2].Id.

FAIL
FAIL    github.com/terraform-providers/terraform-provider-aws/aws       105.334s
make: *** [testacc] Error 1
@bflad bflad added bug Addresses a defect in current functionality. service/cloudwatch Issues and PRs that pertain to the cloudwatch service. labels Mar 28, 2019
@bflad bflad added this to the v2.4.0 milestone Mar 28, 2019
@bflad
Copy link
Contributor

bflad commented Mar 28, 2019

The fix for this has been merged and will release with version 2.4.0 of the Terraform AWS Provider, likely later today.

@bflad
Copy link
Contributor

bflad commented Mar 29, 2019

This has been released in version 2.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@IslamAzab
Copy link

I can replicate the error by:

  • creating a relative metric
  • then trying to update it
Updating metric alarm failed: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutMetricAlarmInput.Metrics[2].Id.
$ terraform -v
Terraform v0.11.13
+ provider.aws v1.60.0

@sjauld
Copy link
Contributor

sjauld commented Jun 13, 2019 via email

@ghost
Copy link

ghost commented Nov 3, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/cloudwatch Issues and PRs that pertain to the cloudwatch service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants