503 error on creating several CloudSQL databases (fails after third was actually created)

[hashibot]

This issue was originally opened by @Blasterdick as hashicorp/terraform#5231. It was migrated here as part of the provider split. The original body of the issue is below.

---

.tf for a CloudSQL instance and it's databases:

resource "google_sql_database_instance" "default" {
    region = "europe-west1"
    database_version = "MYSQL_5_5"
    settings {
        tier = "D1"
        activation_policy = "ON_DEMAND"
        authorized_gae_applications = []
        pricing_plan = "PER_USE"
        replication_type = "SYNCHRONOUS"

        backup_configuration {
            binary_log_enabled = true
            enabled = true
            start_time = "18:00"
        }   

        ip_configuration {
            ipv4_enabled = true
            authorized_networks {
                name = "office_network"
                value = "123.456.789.000"
            }   
        }   

        location_preference {
            zone = "europe-west1-b"
        } 
    }   
}   

resource "google_sql_database" "db-temp-test1" {
    name = "db-temp-test1"
    instance = "${google_sql_database_instance.default.name}"
}

resource "google_sql_database" "db-temp-test2" {
    name = "db-temp-test2"
    instance = "${google_sql_database_instance.default.name}"
}

resource "google_sql_database" "db-temp-test3" {
    name = "db-temp-test3"
    instance = "${google_sql_database_instance.default.name}"
}

Terraform plan sample:

+ google_sql_database.db-temp-test1
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test1"
    self_link: "" => "<computed>"

+ google_sql_database.db-temp-test2
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test2"
    self_link: "" => "<computed>"

+ google_sql_database.db-temp-test3
    instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
    name:      "" => "db-temp-test3"
    self_link: "" => "<computed>"

Terraform apply sample:

google_sql_database.db-temp-test3: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test3"
  self_link: "" => "<computed>"
google_sql_database.db-temp-test1: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test1"
  self_link: "" => "<computed>"
google_sql_database.db-temp-test2: Creating...
  instance:  "" => "terraform-kez4yt4ivzazjcokccu4zw6wni"
  name:      "" => "db-temp-test2"
  self_link: "" => "<computed>"

Error message:

* google_sql_database.db-temp-test2: Error, failed to insert database db-temp-test2 into instance terraform-kez4yt4ivzazjcokccu4zw6wni: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db-temp-test1: Error, failed to insert database db-temp-test1 into instance terraform-kez4yt4ivzazjcokccu4zw6wni: googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_database.db-temp-test3: Error, failure waiting for insertion of db-temp-test3 into terraform-kez4yt4ivzazjcokccu4zw6wni: speckle::MySQL exception: mysql_query You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-temp-test3' at line 1

[nat-henderson]

Good news! No repro, so it must have been fixed at some point. Likely an upstream fix, if I understand the issue correctly.

https://gist.github.com/anonymous/41bc51dc8de1843bdfc6b0b3cd70ac4f

google_sql_database.db-temp-test1: Creation complete after 8s (ID: terraform-20180116211249585700000001:db-temp-test1)
google_sql_database.db-temp-test2: Still creating... (10s elapsed)
google_sql_database.db-temp-test3: Still creating... (10s elapsed)
google_sql_database.db-temp-test2: Creation complete after 15s (ID: terraform-20180116211249585700000001:db-temp-test2)
google_sql_database.db-temp-test3: Still creating... (20s elapsed)
google_sql_database.db-temp-test3: Creation complete after 23s (ID: terraform-20180116211249585700000001:db-temp-test3)

[rowanjacobs]

We're still seeing this issue, with provider version 1.7.0:

google_sql_database.routing: Still creating... (30s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* google_sql_database.routing: 1 error(s) occurred:

* google_sql_database.routing: Error reading SQL Database "routing" in instance "terraforming-gcp-env-rkr6sqohpgi": googleapi: Error 503: Service temporarily unavailable., serverException

[ArchiFleKs]

I'm having the same error today when creating user on a newly created database:

Error: Error applying plan:

1 error(s) occurred:

* google_sql_user.database_user: 1 error(s) occurred:

* google_sql_user.database_user: Error reading SQL User "postgres" in instance "gcp-postgresql-core-test-database": googleapi: Error 503: Service temporarily unavailable., server
Exception

[rosbo]

We added retry logic on 503 errors in #1212. The next release should fix the problem your are experiencing.

[francisd]

we're still experiencing this issue:

Error: Error refreshing state: 7 error(s) occurred:

* google_sql_database.foo: 1 error(s) occurred:

* google_sql_database.foo: google_sql_database.foo: Error reading SQL Database "bar" in instance "foo": googleapi: Error 503: Service temporarily unavailable., serverException
* google_sql_user.foo: 1 error(s) occurred:

* google_sql_user.foo: google_sql_user.foo: Error reading SQL User "bar" in instance "foo": googleapi: Error 503: Service temporarily unavailable., serverException

$ terraform -v
Terraform v0.11.7
+ provider.aws v1.14.1
+ provider.cloudflare v1.0.0
+ provider.google v1.10.0
+ provider.kubernetes v1.1.0
+ provider.template v1.0.0

[nat-henderson]

Ah, shoot, that makes sense if the servers take a really long time to recover. We are retrying, but only until a timeout. Right now it's either 3 minutes or 1 minute, depending on where in the process the read errors come - I'll send a PR to bump them both to 5 minutes, and fingers crossed, that should do it.

[nat-henderson]

I hope that the higher timeouts will be the end of it - if you see more problems after the next release, please comment again.

[baudday]

I'm currently receiving the following error when attempting to run terraform plan

module.database.google_sql_user.mgmt_user: google_sql_user.mgmt_user: Error reading SQL User "mgmt_user" in instance "master-instance": googleapi: Error 503: Service temporarily unavailable. This is most likely a transient error. Please retry., transientError

[nat-henderson]

@baudday Although that's also a 503, it's a different 503 - transientError instead of serverException. Can you open a new issue, including with debug logs?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!