Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating multiple Apigee instances failing #10525

Assignees
Labels
persistent-bug Hard to diagnose or long lived bugs for which resolutions are more like feature work than bug work priority/2

Comments

@matt-hottinger
Copy link

matt-hottinger commented Nov 9, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform: 0.15.5
Provider: hashicorp/google v4.0.0

Affected Resource(s)

  • google_apigee_instance

Terraform Configuration Files

resource "google_kms_key_ring" "apigee_runtime_keyring" {
  project    = var.project_name
  for_each   = toset(var.regions)
  name       = "apigee-runtime-keyring-${each.value}"
  location   = each.value
}

resource "google_kms_crypto_key" "apigee_runtime_key" {
  for_each        = toset(var.regions)
  name            = "apigee-runtime-key-${each.value}"
  key_ring        = google_kms_key_ring.apigee_runtime_keyring[each.value].id
  rotation_period = "2592000"

  lifecycle {
    prevent_destroy = false
  }
}

resource "google_kms_crypto_key_iam_binding" "disk" {
  for_each = toset(var.regions)

  crypto_key_id = google_kms_crypto_key.apigee_runtime_key[each.value].id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"

  members = [
    "serviceAccount:[email protected]"
  ]
}

resource "google_apigee_instance" "apigee_instance" {
  for_each           = toset(var.regions)
  name           = each.value
  location             = each.value
  description            = "Terraform-managed Apigee Runtime Instance"
  display_name           = each.value
  org_id               = format("%s/%s","organizations",var.project_name)
  peering_cidr_range      = "SLASH_20"
  disk_encryption_key_name = google_kms_crypto_key.apigee_runtime_key[each.value].id

  depends_on = [
    google_kms_crypto_key.apigee_runtime_key
  ]
}

Debug Output

Panic Output

Expected Behavior

Both Apigee Instance are created successfully

Actual Behavior

google_apigee_instance.apigee_instance["region1"]: Creating...
google_apigee_instance.apigee_instance["region2"]: Creating...
google_apigee_instance.apigee_instance["region1"]: Still creating... [10s elapsed]
google_apigee_instance.apigee_instance["region1"]: Still creating... [20s elapsed]
google_apigee_instance.apigee_instance["region1"]: Still creating... [30s elapsed]
google_apigee_instance.apigee_instance["region1"]: Still creating... [40s elapsed]
....
google_apigee_instance.apigee_instance["region1"]: Still creating... [23m50s elapsed]
google_apigee_instance.apigee_instance["region1"]: Still creating... [24m0s elapsed]
google_apigee_instance.apigee_instance["region1"]: Still creating... [24m10s elapsed]
google_apigee_instance.apigee_instance["region1"]: Creation complete after 24m19s [id=organizations/my-apigee-org/instances/region1]
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m�[1m�[31mError: �[0m�[0m�[1mError creating Instance: googleapi: Error 400: the resource is locked by another operation: guid-guid-guid-guid
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0mDetails:
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m[
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m  {
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m    "@type": "type.googleapis.com/google.rpc.RequestInfo",
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m    "requestId": "rqId"
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m  }
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m]�[0m
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m�[0m  with google_apigee_instance.apigee_instance["region2"],
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m  on main.tf line 66, in resource "google_apigee_instance" "apigee_instance":
error	01-Nov-2021 14:06:54	�[31m│�[0m �[0m  66: resource "google_apigee_instance" "apigee_instance"

Steps to Reproduce

Attempt to create two Apigee Instances back-to-back.

  1. terraform apply

Important Factoids

References

  • b/218854075
@edwardmedia edwardmedia self-assigned this Nov 9, 2021
@edwardmedia
Copy link
Contributor

edwardmedia commented Nov 12, 2021

@matt-hottinger did you see below error? @melinath what do you see?

googleapi: Error 400: the number of instance cannot exceed the limit 1.

@edwardmedia edwardmedia assigned melinath and unassigned edwardmedia Nov 12, 2021
@matt-hottinger
Copy link
Author

matt-hottinger commented Nov 12, 2021

@matt-hottinger did you see below error? @melinath what do you see?

googleapi: Error 400: the number of instance cannot exceed the limit 1.

I think that makes sense in context of a single GCP region. Our use case would serially stand up a single Apigee instances in two distinct regions, which would respect the limit mentioned above.

@matt-hottinger
Copy link
Author

Checkout this bug that was reported with the google_apigee_instance_attachment resource. I believe this bug is related in terms of root cause given the presentation of the error: #10084

Hopefully the additional context helps.

@matt-hottinger
Copy link
Author

@edwardmedia / @melinath, has there been anymore investigation on this issue or do you need me to provide more information?

@melinath
Copy link
Collaborator

Sorry, I wasn't able to do further investigation at this time. I'll remove my assignment so that it gets flagged as an unassigned bug.

@rileykarson
Copy link
Collaborator

rileykarson commented Feb 17, 2022

@xuchenma's change in GoogleCloudPlatform/magic-modules#5701 addressed the primary issue, I'll add an additional prevention mechanism (a retry rule) and close. These changes may not eliminate the issue entirely, as the restriction is still present at the API layer, but should substantially reduce the chances of it occurring.

@c2thorn c2thorn assigned c2thorn and unassigned rileykarson Mar 28, 2022
@c2thorn c2thorn added the persistent-bug Hard to diagnose or long lived bugs for which resolutions are more like feature work than bug work label Mar 28, 2022
@c2thorn c2thorn added this to the Sprint 7 (2022) milestone Mar 28, 2022
@matihost
Copy link

matihost commented Apr 1, 2022

Same problem when you add SINGLE env as attachment to two or more Apigee instances:

"  # module.someModuleName.google_apigee_instance_attachment.instance-env-attachment[\\"region1\\"] will be created", 
"  + resource \\"google_apigee_instance_attachment\\" \\"instance-env-attachment\\" {", 
"      + environment = \\"envName\\"", 
"      + id          = (known after apply)", 
"      + instance_id = \\"organizations/orgname/instances/region1\\"", 
"      + name        = (known after apply)", 
"    }", 
"", 
"  # module.someModuleName.google_apigee_instance_attachment.instance-env-attachment[\\"region2\\"] will be created", 
"  + resource \\"google_apigee_instance_attachment\\" \\"instance-env-attachment\\" {", 
"      + environment = \\"envName\\"", 
"      + id          = (known after apply)", 
"      + instance_id = \\"organizations/orgname/instances/region2\\"", 
"      + name        = (known after apply)", 
"    }", 

Terraform see that it can create these two objects in parallel - but Apigee API does not allow it.
And GCP Terraform Code does not prevent from it.

So it is more generic problem in Terraform GCP Provider for APigeeX object creations.

@github-actions
Copy link

github-actions bot commented Sep 9, 2022

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.