Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TERRAFORM CRASH: err="connection is shut down" #30

Open
sandnabba opened this issue Dec 12, 2019 · 4 comments
Open

TERRAFORM CRASH: err="connection is shut down" #30

sandnabba opened this issue Dec 12, 2019 · 4 comments
Labels
bug Something isn't working

Comments

@sandnabba
Copy link

Today, when applying some changes, Terraform crashed with this message:

emil@DvorakWork: ~/git/ops/sre-metrocard/estate/threatmonitor/terraform-templates/appoptics $> terraform apply
appoptics_alert.replica_lag: Refreshing state... (ID: 61427079)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418689)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418687)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418688)
appoptics_alert.cpu_usage: Refreshing state... (ID: 61427077)
appoptics_alert.sqs_incorrect_format: Refreshing state... (ID: 61418674)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418679)
appoptics_alert.cpu_usage: Refreshing state... (ID: 61424432)
appoptics_alert.replica_lag: Refreshing state... (ID: 61424219)
appoptics_alert.DMLLatency: Refreshing state... (ID: 61427074)
appoptics_alert.FreeLocalStorage: Refreshing state... (ID: 61426611)
appoptics_alert.sqs_age_of_oldest_message: Refreshing state... (ID: 61418692)
appoptics_alert.replica_lag: Refreshing state... (ID: 61427073)
appoptics_alert.DMLLatency: Refreshing state... (ID: 61426610)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418684)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418690)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418678)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418685)
appoptics_alert.DMLLatency: Refreshing state... (ID: 61427078)
appoptics_alert.cpu_usage: Refreshing state... (ID: 61427075)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418677)
appoptics_alert.sqs_incorrect_format: Refreshing state... (ID: 61418693)
appoptics_alert.sqs_age_of_oldest_message: Refreshing state... (ID: 61418676)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418675)
appoptics_alert.replica_lag: Refreshing state... (ID: 61424218)
appoptics_alert.FreeLocalStorage: Refreshing state... (ID: 61426609)
appoptics_alert.cpu_usage: Refreshing state... (ID: 61424431)
appoptics_alert.FreeLocalStorage: Refreshing state... (ID: 61427072)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418672)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418673)
appoptics_alert.DMLLatency: Refreshing state... (ID: 61426612)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418683)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418694)
appoptics_alert.FreeLocalStorage: Refreshing state... (ID: 61427076)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418691)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418681)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418680)
appoptics_alert.lb_http_5xx: Refreshing state... (ID: 61418682)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418686)
appoptics_alert.app_http_5xx: Refreshing state... (ID: 61418671)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.prod-general.module.shared-rds-aurora.appoptics_alert.DMLLatency
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""
      condition.2447241622.detect_reset:     "false" => "false"
      condition.2447241622.duration:         "300" => "0"
      condition.2447241622.metric_name:      "AWS.RDS.DMLLatency" => ""
      condition.2447241622.summary_function: "max" => ""
      condition.2447241622.tag.#:            "2" => "0"
      condition.2447241622.tag.0.grouped:    "true" => "false"
      condition.2447241622.tag.0.name:       "awsaccount" => ""
      condition.2447241622.tag.0.values.#:   "1" => "0"
      condition.2447241622.tag.0.values.0:   "tm-prod" => ""
      condition.2447241622.tag.1.grouped:    "true" => "false"
      condition.2447241622.tag.1.name:       "dbclusteridentifier" => ""
      condition.2447241622.tag.1.values.#:   "1" => "0"
      condition.2447241622.tag.1.values.0:   "*prod*" => ""
      condition.2447241622.threshold:        "1" => "0"
      condition.2447241622.type:             "above" => ""
      condition.3401310160.detect_reset:     "" => ""
      condition.3401310160.duration:         "" => "300"
      condition.3401310160.metric_name:      "" => "AWS.RDS.DMLLatency"
      condition.3401310160.summary_function: "" => "max"
      condition.3401310160.tag.#:            "0" => "2"
      condition.3401310160.tag.0.grouped:    "" => "true"
      condition.3401310160.tag.0.name:       "" => "awsaccount"
      condition.3401310160.tag.0.values.#:   "0" => "1"
      condition.3401310160.tag.0.values.0:   "" => "tm-prod"
      condition.3401310160.tag.1.grouped:    "" => "true"
      condition.3401310160.tag.1.name:       "" => "dbclusteridentifier"
      condition.3401310160.tag.1.values.#:   "0" => "1"
      condition.3401310160.tag.1.values.0:   "" => "*tm-db*"
      condition.3401310160.threshold:        "" => "1"
      condition.3401310160.type:             "" => "above"
      name:                                  "prod-prod-AWS.RDS.DMLLatency" => "prod-tm-db-AWS.RDS.DMLLatency"

  ~ module.prod-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""
      condition.1108553706.detect_reset:     "false" => "false"
      condition.1108553706.duration:         "60" => "0"
      condition.1108553706.metric_name:      "AWS.RDS.FreeLocalStorage" => ""
      condition.1108553706.summary_function: "" => ""
      condition.1108553706.tag.#:            "2" => "0"
      condition.1108553706.tag.0.grouped:    "true" => "false"
      condition.1108553706.tag.0.name:       "awsaccount" => ""
      condition.1108553706.tag.0.values.#:   "1" => "0"
      condition.1108553706.tag.0.values.0:   "tm-prod" => ""
      condition.1108553706.tag.1.grouped:    "true" => "false"
      condition.1108553706.tag.1.name:       "dbclusteridentifier" => ""
      condition.1108553706.tag.1.values.#:   "1" => "0"
      condition.1108553706.tag.1.values.0:   "*prod*" => ""
      condition.1108553706.threshold:        "5000000000" => "0"
      condition.1108553706.type:             "below" => ""
      condition.2668980972.detect_reset:     "" => ""
      condition.2668980972.duration:         "" => "60"
      condition.2668980972.metric_name:      "" => "AWS.RDS.FreeLocalStorage"
      condition.2668980972.summary_function: "" => ""
      condition.2668980972.tag.#:            "0" => "2"
      condition.2668980972.tag.0.grouped:    "" => "true"
      condition.2668980972.tag.0.name:       "" => "awsaccount"
      condition.2668980972.tag.0.values.#:   "0" => "1"
      condition.2668980972.tag.0.values.0:   "" => "tm-prod"
      condition.2668980972.tag.1.grouped:    "" => "true"
      condition.2668980972.tag.1.name:       "" => "dbclusteridentifier"
      condition.2668980972.tag.1.values.#:   "0" => "1"
      condition.2668980972.tag.1.values.0:   "" => "*tm-db*"
      condition.2668980972.threshold:        "" => "5000000000"
      condition.2668980972.type:             "" => "below"
      name:                                  "prod-prod-AWS.RDS.FreeLocalStorage" => "prod-tm-db-AWS.RDS.FreeLocalStorage"

  ~ module.prod-general.module.shared-rds-aurora.appoptics_alert.cpu_usage
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""
      condition.2902886980.detect_reset:     "" => ""
      condition.2902886980.duration:         "" => "300"
      condition.2902886980.metric_name:      "" => "AWS.RDS.CPUUtilization"
      condition.2902886980.summary_function: "" => "max"
      condition.2902886980.tag.#:            "0" => "2"
      condition.2902886980.tag.0.grouped:    "" => "true"
      condition.2902886980.tag.0.name:       "" => "awsaccount"
      condition.2902886980.tag.0.values.#:   "0" => "1"
      condition.2902886980.tag.0.values.0:   "" => "tm-prod"
      condition.2902886980.tag.1.grouped:    "" => "true"
      condition.2902886980.tag.1.name:       "" => "dbclusteridentifier"
      condition.2902886980.tag.1.values.#:   "0" => "1"
      condition.2902886980.tag.1.values.0:   "" => "*tm-db*"
      condition.2902886980.threshold:        "" => "50"
      condition.2902886980.type:             "" => "above"
      condition.2997775536.detect_reset:     "false" => "false"
      condition.2997775536.duration:         "300" => "0"
      condition.2997775536.metric_name:      "AWS.RDS.CPUUtilization" => ""
      condition.2997775536.summary_function: "max" => ""
      condition.2997775536.tag.#:            "2" => "0"
      condition.2997775536.tag.0.grouped:    "true" => "false"
      condition.2997775536.tag.0.name:       "awsaccount" => ""
      condition.2997775536.tag.0.values.#:   "1" => "0"
      condition.2997775536.tag.0.values.0:   "tm-prod" => ""
      condition.2997775536.tag.1.grouped:    "true" => "false"
      condition.2997775536.tag.1.name:       "dbclusteridentifier" => ""
      condition.2997775536.tag.1.values.#:   "1" => "0"
      condition.2997775536.tag.1.values.0:   "*prod*" => ""
      condition.2997775536.threshold:        "50" => "0"
      condition.2997775536.type:             "above" => ""
      name:                                  "prod-prod-AWS.RDS.CPUUtilization" => "prod-tm-db-AWS.RDS.CPUUtilization"

  ~ module.prod-general.module.shared-rds-aurora.appoptics_alert.replica_lag
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""
      condition.1187382735.detect_reset:     "false" => "false"
      condition.1187382735.duration:         "60" => "0"
      condition.1187382735.metric_name:      "AWS.RDS.AuroraReplicaLagMaximum" => ""
      condition.1187382735.summary_function: "max" => ""
      condition.1187382735.tag.#:            "2" => "0"
      condition.1187382735.tag.0.grouped:    "true" => "false"
      condition.1187382735.tag.0.name:       "awsaccount" => ""
      condition.1187382735.tag.0.values.#:   "1" => "0"
      condition.1187382735.tag.0.values.0:   "tm-prod" => ""
      condition.1187382735.tag.1.grouped:    "true" => "false"
      condition.1187382735.tag.1.name:       "dbclusteridentifier" => ""
      condition.1187382735.tag.1.values.#:   "1" => "0"
      condition.1187382735.tag.1.values.0:   "*prod*" => ""
      condition.1187382735.threshold:        "50" => "0"
      condition.1187382735.type:             "above" => ""
      condition.18315514.detect_reset:       "" => ""
      condition.18315514.duration:           "" => "60"
      condition.18315514.metric_name:        "" => "AWS.RDS.AuroraReplicaLagMaximum"
      condition.18315514.summary_function:   "" => "max"
      condition.18315514.tag.#:              "0" => "2"
      condition.18315514.tag.0.grouped:      "" => "true"
      condition.18315514.tag.0.name:         "" => "awsaccount"
      condition.18315514.tag.0.values.#:     "0" => "1"
      condition.18315514.tag.0.values.0:     "" => "tm-prod"
      condition.18315514.tag.1.grouped:      "" => "true"
      condition.18315514.tag.1.name:         "" => "dbclusteridentifier"
      condition.18315514.tag.1.values.#:     "0" => "1"
      condition.18315514.tag.1.values.0:     "" => "*tm-db*"
      condition.18315514.threshold:          "" => "50"
      condition.18315514.type:               "" => "above"
      name:                                  "prod-prod-AWS.RDS.AuroraReplicaLagMaximum" => "prod-tm-db-AWS.RDS.AuroraReplicaLagMaximum"

  ~ module.staging-general.module.shared-rds-aurora.appoptics_alert.DMLLatency
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""

  ~ module.staging-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""

  ~ module.staging-general.module.shared-rds-aurora.appoptics_alert.cpu_usage
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""

  ~ module.staging-general.module.shared-rds-aurora.appoptics_alert.replica_lag
      attributes.%:                          "0" => "1"
      attributes.runbook_url:                "" => ""


Plan: 0 to add, 8 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.staging-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: Modifying... (ID: 61427078)
  attributes.%:           "0" => "1"
  attributes.runbook_url: "" => ""
module.prod-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: Modifying... (ID: 61427076)
  attributes.%:                          "0" => "1"
  attributes.runbook_url:                "" => ""
  condition.1108553706.detect_reset:     "false" => "false"
  condition.1108553706.duration:         "60" => "0"
  condition.1108553706.metric_name:      "AWS.RDS.FreeLocalStorage" => ""
  condition.1108553706.summary_function: "" => ""
  condition.1108553706.tag.#:            "2" => "0"
  condition.1108553706.tag.0.grouped:    "true" => "false"
  condition.1108553706.tag.0.name:       "awsaccount" => ""
  condition.1108553706.tag.0.values.#:   "1" => "0"
  condition.1108553706.tag.0.values.0:   "tm-prod" => ""
  condition.1108553706.tag.1.grouped:    "true" => "false"
  condition.1108553706.tag.1.name:       "dbclusteridentifier" => ""
  condition.1108553706.tag.1.values.#:   "1" => "0"
  condition.1108553706.tag.1.values.0:   "*prod*" => ""
  condition.1108553706.threshold:        "5000000000" => "0"
  condition.1108553706.type:             "below" => ""
  condition.2668980972.detect_reset:     "" => ""
  condition.2668980972.duration:         "" => "60"
  condition.2668980972.metric_name:      "" => "AWS.RDS.FreeLocalStorage"
  condition.2668980972.summary_function: "" => ""
  condition.2668980972.tag.#:            "0" => "2"
  condition.2668980972.tag.0.grouped:    "" => "true"
  condition.2668980972.tag.0.name:       "" => "awsaccount"
  condition.2668980972.tag.0.values.#:   "0" => "1"
  condition.2668980972.tag.0.values.0:   "" => "tm-prod"
  condition.2668980972.tag.1.grouped:    "" => "true"
  condition.2668980972.tag.1.name:       "" => "dbclusteridentifier"
  condition.2668980972.tag.1.values.#:   "0" => "1"
  condition.2668980972.tag.1.values.0:   "" => "*tm-db*"
  condition.2668980972.threshold:        "" => "5000000000"
  condition.2668980972.type:             "" => "below"
  name:                                  "prod-prod-AWS.RDS.FreeLocalStorage" => "prod-tm-db-AWS.RDS.FreeLocalStorage"
module.prod-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: Modifying... (ID: 61427077)
  attributes.%:                          "0" => "1"
  attributes.runbook_url:                "" => ""
  condition.2902886980.detect_reset:     "" => ""
  condition.2902886980.duration:         "" => "300"
  condition.2902886980.metric_name:      "" => "AWS.RDS.CPUUtilization"
  condition.2902886980.summary_function: "" => "max"
  condition.2902886980.tag.#:            "0" => "2"
  condition.2902886980.tag.0.grouped:    "" => "true"
  condition.2902886980.tag.0.name:       "" => "awsaccount"
  condition.2902886980.tag.0.values.#:   "0" => "1"
  condition.2902886980.tag.0.values.0:   "" => "tm-prod"
  condition.2902886980.tag.1.grouped:    "" => "true"
  condition.2902886980.tag.1.name:       "" => "dbclusteridentifier"
  condition.2902886980.tag.1.values.#:   "0" => "1"
  condition.2902886980.tag.1.values.0:   "" => "*tm-db*"
  condition.2902886980.threshold:        "" => "50"
  condition.2902886980.type:             "" => "above"
  condition.2997775536.detect_reset:     "false" => "false"
  condition.2997775536.duration:         "300" => "0"
  condition.2997775536.metric_name:      "AWS.RDS.CPUUtilization" => ""
  condition.2997775536.summary_function: "max" => ""
  condition.2997775536.tag.#:            "2" => "0"
  condition.2997775536.tag.0.grouped:    "true" => "false"
  condition.2997775536.tag.0.name:       "awsaccount" => ""
  condition.2997775536.tag.0.values.#:   "1" => "0"
  condition.2997775536.tag.0.values.0:   "tm-prod" => ""
  condition.2997775536.tag.1.grouped:    "true" => "false"
  condition.2997775536.tag.1.name:       "dbclusteridentifier" => ""
  condition.2997775536.tag.1.values.#:   "1" => "0"
  condition.2997775536.tag.1.values.0:   "*prod*" => ""
  condition.2997775536.threshold:        "50" => "0"
  condition.2997775536.type:             "above" => ""
  name:                                  "prod-prod-AWS.RDS.CPUUtilization" => "prod-tm-db-AWS.RDS.CPUUtilization"
module.staging-general.module.shared-rds-aurora.appoptics_alert.replica_lag: Modifying... (ID: 61427079)
  attributes.%:           "0" => "1"
  attributes.runbook_url: "" => ""
module.staging-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: Modifying... (ID: 61427072)
  attributes.%:           "0" => "1"
  attributes.runbook_url: "" => ""
module.staging-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: Modifying... (ID: 61427075)
  attributes.%:           "0" => "1"
  attributes.runbook_url: "" => ""
module.prod-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: Modifying... (ID: 61427074)
  attributes.%:                          "0" => "1"
  attributes.runbook_url:                "" => ""
  condition.2447241622.detect_reset:     "false" => "false"
  condition.2447241622.duration:         "300" => "0"
  condition.2447241622.metric_name:      "AWS.RDS.DMLLatency" => ""
  condition.2447241622.summary_function: "max" => ""
  condition.2447241622.tag.#:            "2" => "0"
  condition.2447241622.tag.0.grouped:    "true" => "false"
  condition.2447241622.tag.0.name:       "awsaccount" => ""
  condition.2447241622.tag.0.values.#:   "1" => "0"
  condition.2447241622.tag.0.values.0:   "tm-prod" => ""
  condition.2447241622.tag.1.grouped:    "true" => "false"
  condition.2447241622.tag.1.name:       "dbclusteridentifier" => ""
  condition.2447241622.tag.1.values.#:   "1" => "0"
  condition.2447241622.tag.1.values.0:   "*prod*" => ""
  condition.2447241622.threshold:        "1" => "0"
  condition.2447241622.type:             "above" => ""
  condition.3401310160.detect_reset:     "" => ""
  condition.3401310160.duration:         "" => "300"
  condition.3401310160.metric_name:      "" => "AWS.RDS.DMLLatency"
  condition.3401310160.summary_function: "" => "max"
  condition.3401310160.tag.#:            "0" => "2"
  condition.3401310160.tag.0.grouped:    "" => "true"
  condition.3401310160.tag.0.name:       "" => "awsaccount"
  condition.3401310160.tag.0.values.#:   "0" => "1"
  condition.3401310160.tag.0.values.0:   "" => "tm-prod"
  condition.3401310160.tag.1.grouped:    "" => "true"
  condition.3401310160.tag.1.name:       "" => "dbclusteridentifier"
  condition.3401310160.tag.1.values.#:   "0" => "1"
  condition.3401310160.tag.1.values.0:   "" => "*tm-db*"
  condition.3401310160.threshold:        "" => "1"
  condition.3401310160.type:             "" => "above"
  name:                                  "prod-prod-AWS.RDS.DMLLatency" => "prod-tm-db-AWS.RDS.DMLLatency"
module.prod-general.module.shared-rds-aurora.appoptics_alert.replica_lag: Modifying... (ID: 61427073)
  attributes.%:                          "0" => "1"
  attributes.runbook_url:                "" => ""
  condition.1187382735.detect_reset:     "false" => "false"
  condition.1187382735.duration:         "60" => "0"
  condition.1187382735.metric_name:      "AWS.RDS.AuroraReplicaLagMaximum" => ""
  condition.1187382735.summary_function: "max" => ""
  condition.1187382735.tag.#:            "2" => "0"
  condition.1187382735.tag.0.grouped:    "true" => "false"
  condition.1187382735.tag.0.name:       "awsaccount" => ""
  condition.1187382735.tag.0.values.#:   "1" => "0"
  condition.1187382735.tag.0.values.0:   "tm-prod" => ""
  condition.1187382735.tag.1.grouped:    "true" => "false"
  condition.1187382735.tag.1.name:       "dbclusteridentifier" => ""
  condition.1187382735.tag.1.values.#:   "1" => "0"
  condition.1187382735.tag.1.values.0:   "*prod*" => ""
  condition.1187382735.threshold:        "50" => "0"
  condition.1187382735.type:             "above" => ""
  condition.18315514.detect_reset:       "" => ""
  condition.18315514.duration:           "" => "60"
  condition.18315514.metric_name:        "" => "AWS.RDS.AuroraReplicaLagMaximum"
  condition.18315514.summary_function:   "" => "max"
  condition.18315514.tag.#:              "0" => "2"
  condition.18315514.tag.0.grouped:      "" => "true"
  condition.18315514.tag.0.name:         "" => "awsaccount"
  condition.18315514.tag.0.values.#:     "0" => "1"
  condition.18315514.tag.0.values.0:     "" => "tm-prod"
  condition.18315514.tag.1.grouped:      "" => "true"
  condition.18315514.tag.1.name:         "" => "dbclusteridentifier"
  condition.18315514.tag.1.values.#:     "0" => "1"
  condition.18315514.tag.1.values.0:     "" => "*tm-db*"
  condition.18315514.threshold:          "" => "50"
  condition.18315514.type:               "" => "above"
  name:                                  "prod-prod-AWS.RDS.AuroraReplicaLagMaximum" => "prod-tm-db-AWS.RDS.AuroraReplicaLagMaximum"

Error: Error applying plan:

8 error(s) occurred:

* module.prod-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
* module.staging-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
* module.staging-general.module.shared-rds-aurora.appoptics_alert.replica_lag: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
* module.prod-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF
* module.staging-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
* module.prod-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
* module.prod-general.module.shared-rds-aurora.appoptics_alert.replica_lag: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
* module.staging-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


panic: interface conversion: interface {} is map[string]interface {}, not []interface {}
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: 
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: goroutine 328 [running]:
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: github.com/appoptics/terraform-provider-appoptics/appoptics.resourceAppOpticsAlertUpdate(0xc00027caf0, 0xfba2c0, 0xc00039a7e0, 0x24, 0x1a30800)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /home/emil/git/go/src/github.com/appoptics/terraform-provider-appoptics/appoptics/resource_appoptics_alert.go:464 +0x14ed
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc0000eb110, 0xc0003e0c30, 0xc000762c60, 0xfba2c0, 0xc00039a7e0, 0x1, 0xc0006a0ba8, 0x40db56)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /home/emil/git/go/src/github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:231 +0x251
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc0000eb1f0, 0xc0003e0b40, 0xc0003e0c30, 0xc000762c60, 0xc0007c7b60, 0xc0004d6600, 0x18)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /home/emil/git/go/src/github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0x99
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc0001f3240, 0xc000762800, 0xc0007e1a20, 0x0, 0x0)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /home/emil/git/go/src/github.com/appoptics/terraform-provider-appoptics/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x57
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: reflect.Value.call(0xc0004c23c0, 0xc000010d88, 0x13, 0x101c652, 0x4, 0xc0006a0f18, 0x3, 0x3, 0x0, 0x0, ...)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /usr/lib/go-1.13/src/reflect/value.go:460 +0x5f6
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: reflect.Value.Call(0xc0004c23c0, 0xc000010d88, 0x13, 0xc000617718, 0x3, 0x3, 0x0, 0x0, 0x0)
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics:       /usr/lib/go-1.13/src/reflect/value.go:321 +0xb4
2019-12-12T14:51:12.196+0100 [DEBUG] plugin.terraform-provider-appoptics: net/rpc.(*service).call(0xc00042e5c0, 0xc00047b2c0, 0xc00059ca60, 0xc00059ca70, 0xc000446180, 0xc0001f7c20, 0xe22060, 0xc000762800, 0x16, 0xe220a0, ...)
2019-12-12T14:51:12.197+0100 [DEBUG] plugin.terraform-provider-appoptics:       /usr/lib/go-1.13/src/net/rpc/server.go:377 +0x16f
2019-12-12T14:51:12.197+0100 [DEBUG] plugin.terraform-provider-appoptics: created by net/rpc.(*Server).ServeCodec
2019-12-12T14:51:12.197+0100 [DEBUG] plugin.terraform-provider-appoptics:       /usr/lib/go-1.13/src/net/rpc/server.go:474 +0x42b
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019-12-12T14:51:12.198+0100 [DEBUG] plugin: plugin process exited: path=/home/emil/.terraform.d/plugins/terraform-provider-appoptics
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.prod-general.module.shared-rds-aurora.appoptics_alert.cpu_usage
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.cpu_usage: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.staging-general.module.shared-rds-aurora.appoptics_alert.cpu_usage
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.staging-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.prod-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.prod-general.module.shared-rds-aurora.appoptics_alert.replica_lag
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyProvisioners
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalIf
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteState
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.staging-general.module.shared-rds-aurora.appoptics_alert.DMLLatency
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalWriteDiff
2019/12/12 14:51:12 [TRACE] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.staging-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.replica_lag: unexpected EOF
2019/12/12 14:51:12 [ERROR] root.prod-general.shared-rds-aurora: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* appoptics_alert.DMLLatency: unexpected EOF
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.prod-general.module.shared-rds-aurora.appoptics_alert.DMLLatency
2019/12/12 14:51:12 [TRACE] [walkApply] Exiting eval tree: module.staging-general.module.shared-rds-aurora.appoptics_alert.replica_lag
2019/12/12 14:51:12 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"
2019/12/12 14:51:12 [TRACE] dag/walk: upstream errored, not walking "provider.appoptics (close)"
2019/12/12 14:51:12 [TRACE] dag/walk: upstream errored, not walking "root"
2019/12/12 14:51:12 [TRACE] Preserving existing state lineage "a4d5e2c1-7415-47ba-abfd-33b7c8d1f104"
2019/12/12 14:51:12 [TRACE] Preserving existing state lineage "a4d5e2c1-7415-47ba-abfd-33b7c8d1f104"
2019/12/12 14:51:12 [TRACE] Preserving existing state lineage "a4d5e2c1-7415-47ba-abfd-33b7c8d1f104"
2019/12/12 14:51:12 [TRACE] Preserving existing state lineage "a4d5e2c1-7415-47ba-abfd-33b7c8d1f104"
2019/12/12 14:51:12 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-12-12T14:51:12.238+0100 [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Efter running apply again, it now crashes with:

Error: Error applying plan:

8 error(s) occurred:

* module.prod-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: 1 error(s) occurred:

* appoptics_alert.DMLLatency: Error creating AppOptics alert prod-tm-db-AWS.RDS.DMLLatency: 
* module.staging-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: 1 error(s) occurred:

* appoptics_alert.cpu_usage: Error creating AppOptics alert staging-staging-AWS.RDS.CPUUtilization: 
* module.prod-general.module.shared-rds-aurora.appoptics_alert.cpu_usage: 1 error(s) occurred:

* appoptics_alert.cpu_usage: Error creating AppOptics alert prod-tm-db-AWS.RDS.CPUUtilization: 
* module.staging-general.module.shared-rds-aurora.appoptics_alert.DMLLatency: 1 error(s) occurred:

* appoptics_alert.DMLLatency: Error creating AppOptics alert staging-staging-AWS.RDS.DMLLatency: 
* module.prod-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: Error creating AppOptics alert prod-tm-db-AWS.RDS.FreeLocalStorage: 
* module.staging-general.module.shared-rds-aurora.appoptics_alert.FreeLocalStorage: 1 error(s) occurred:

* appoptics_alert.FreeLocalStorage: Error creating AppOptics alert staging-staging-AWS.RDS.FreeLocalStorage: 
* module.prod-general.module.shared-rds-aurora.appoptics_alert.replica_lag: 1 error(s) occurred:

* appoptics_alert.replica_lag: Error creating AppOptics alert prod-tm-db-AWS.RDS.AuroraReplicaLagMaximum: 
* module.staging-general.module.shared-rds-aurora.appoptics_alert.replica_lag: 1 error(s) occurred:

* appoptics_alert.replica_lag: Error creating AppOptics alert staging-staging-AWS.RDS.AuroraReplicaLagMaximum: 

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I think that the crashes in because it's trying to create an alert that already exists.

I'll try to clean up my state to see if I can reproduce this error.
@solarchad
Copy link
Contributor

Thanks for sending. Sorry it crashed, that's not ideal. I see that some alert conditions are being set to to have a threshold of "0" which we figured out would fail due to a bug in the Go client but was fixed in here: appoptics/appoptics-api-go#70

I think we should probably update the client dependency to the latest version and perhaps re-run. If you wanted to give it a shot, you can run dep ensure -update github.com/appoptics/appoptics-api-go to upgrade that dependency and then recompile.

@sandnabba
Copy link
Author

sandnabba commented Dec 16, 2019 •
edited
Loading

Hi again!

I actually don't know why it looks like the threshold is 0. It is not, and has never been. The alert has been created by Terraform in the first place, so it should never have been able to be created with that value.
Could it be some kind of default value in any way? Like defaulting to 0 / false if there is no value set?
I would also assume that the state might be somewhat broken, since Terraform crashed.

Anyways, Terraform has been crashing several times now, and It's actually causing so many issues that I'm unable to work with it anymore.
It seems to be related to how many alerts I have defined.

1.

I just tried to change the alert condition for 24 alerts. They where all setup correctly, and a terraform plan showed no changes.
Then I did a small change in the alert condition (duration 600 -> 60), and applied.

2.

On the first run, it successfully changed the condition of 11 alerts before crashing.
The next plan said:
Plan: 0 to add, 13 to change, 0 to destroy

3.

After applying the above change, terraform crashed almost instantly.
The next plan said:
Plan: 22 to add, 0 to change, 0 to destroy.

And now the apply fails for every alert:

Error: Error applying plan:

22 error(s) occurred:

It is failing because all of the alerts are already exists in AO. So in some way the alerts has been removed from the state.
However, it is not crashing anymore. So the crash seems to be related to doing a lot of actual changes at the same time?

@trevrosen
Copy link
Contributor

Hey guys I'm just now seeing this thread (was at Tech Summit in Brno all last week and basically didn't check on anything else). We've landed the PR mentioned above. I'm going to be doing dev on this thing all week this week, so I may reach out with some questions.

@trevrosen
Copy link
Contributor

@sandnabba FYI in Go, you can't have un-initialized memory (for safety reasons), so every type (and therefore every member of a struct) has a "zero value". The zero value of any Int type is, well, 0. That's what's going on here.

@trevrosen trevrosen added the bug Something isn't working label Dec 16, 2019
@trevrosen trevrosen self-assigned this Dec 16, 2019
@trevrosen trevrosen removed their assignment Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@trevrosen @sandnabba @solarchad