CloudSQL behavior change v3.44.0

[jimmydurkin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

terraform -v
Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/google v3.44.0
+ provider registry.terraform.io/hashicorp/http v2.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0

Affected Resource(s)

• google_sql_database_instance

Terraform Configuration Files

resource "google_sql_database_instance" "test" {
  project          = var.project_id
  name             = "testdb"
  database_version = "POSTGRES_12"
  region           = var.region 

  settings {
    ip_configuration {
      ipv4_enabled    = false
      private_network = var.shared_network_self_link
    }
    tier = "db-g1-small"

    disk_size       = 10
    disk_type       = "PD_SSD"

  }
}

provider "google" {
  version = "3.44.0"
}

Debug Output

https://gist.github.com/jimmydurkin/1e2eb317ea1ed7b220d7bfca785d8985

Panic Output

Expected Behavior

This code snippet works on 3.43 and provisions our database

Actual Behavior

After accepting the plan an error is thrown

Error: Failed to retrieve project, pid: <project-id>, err: googleapi: Error 403: The caller does not have permission, forbidden

Steps to Reproduce

1. terraform apply

Important Factoids

This service account that runs this belongs to a different project than the shared vpc. It has minimal permissions.

References

[rileykarson]

It seems probable that the service networking check from GoogleCloudPlatform/magic-modules#3954 is overtuned.

[upodroid]

Hmm

That is very unlikely to happen if you are using Shared VPC. That error appears if the identity running terraform is missing resourcemanager.projects.get permission which unlikely because Compute Network User, Network Admin are roles you should have which include this permission and the second one is needed to create the Service Networking resources to enable CloudSQL to work.

https://cloud.google.com/sql/docs/mysql/private-ip#api_and_iam_requirements

That first err check should have a been less aggresive.

[jimmydurkin]

As far as I can tell the identity running this code only has roles/browser on the shared vpc host project and roles/owner within the service project. In our case the host project is responsible for provisioning the service networking resources and that terraform code runs with a different GCP identity which has those roles.

I am early in my GCP learning so it is likely I've laid this out in an unexpected way, but I still would not expect the usage to change in this way given the version bump.

Please let me know if I must expand IAM permissions to ensure this works smoothly or this behavior changed will be addressed in the provider. Thanks!

[rileykarson]

@jimmydurkin: I've got a fix, but I'd like to test it against this exact case before submitting it. Does this reproduction sound right?

You're using Terraform with some service account foo. foo has roles/browser and no other roles on a VPC host project holding a shared VPC. foo additionally has roles/owner on a service project, where your DB is being created.

It's my understanding foo also needs to be granted using permissions on the host project or on the host network- am I missing a role somewhere?

[upodroid]

If foo has browser role then that error shouldn't appear.

https://cloud.google.com/resource-manager/reference/rest/v1/projects/get

 root  DESKTOP-7JUP8RO  ~  #  gcloud iam roles describe roles/browser
description: Access to browse GCP resources.
etag: AA==
includedPermissions:
- resourcemanager.folders.get
- resourcemanager.folders.list
- resourcemanager.organizations.get
- resourcemanager.projects.get
- resourcemanager.projects.getIamPolicy
- resourcemanager.projects.list
name: roles/browser
stage: GA
title: Browser

[upodroid]

https://gist.github.com/jimmydurkin/1e2eb317ea1ed7b220d7bfca785d8985#file-gistfile1-txt-L294 The GET call shouldn't fail if the identity has browser on the redacted project.

Please double check the permissions.

[rileykarson]

@upodroid: We still want to reduce the permissions required regardless- requiring additional permissions is a breaking change. I believe it was possible in prior versions to only have compute.networkUser granted on the host project.

[upodroid]

The fix would be to change https://github.com/hashicorp/terraform-provider-google/blob/master/google/resource_sql_database_instance.go#L660 660 to match the same as 666

When I wrote that, I assumed that the identity would atleast have resourcemanager.projects.get on the Shared VPC.

[jimmydurkin]

Sorry. yes @upodroid you are correct, the service project owner identity that we use to run this code has no roles in the host vpc project.

i found this command so if it can be trusted there are no permissions granted to this user in the networking host project.

gcloud beta asset search-all-iam-policies --scope=organizations/orgid --query="[email protected]"

---
policy:
  bindings:
  - members:
    - serviceAccount:[email protected]
    role: roles/cloudbuild.builds.editor
  - members:
    - serviceAccount:[email protected]
    role: roles/owner
  - members:
    - serviceAccount:[email protected]
    role: roles/viewer
project: projects/cloudbuild-project
resource: //cloudresourcemanager.googleapis.com/projects/cloudbuild-project
---
policy:
  bindings:
  - members:
    - serviceAccount:[email protected]
    role: roles/owner
project: projects/service-project
resource: //cloudresourcemanager.googleapis.com/projects/service-project
---
policy:
  bindings:
  - members:
    - serviceAccount:[email protected]
    role: roles/browser
project: projects/gke-host-project
resource: //cloudresourcemanager.googleapis.com/projects/gke-host-project
---
policy:
  bindings:
  - members:
    - group:[email protected]
    role: roles/iam.serviceAccountTokenCreator
project: projects/service-project
resource: //iam.googleapis.com/projects/service-project/serviceAccounts/[email protected]

[upodroid]

So how do you create VMs in the service project using the network from Shared VPC?

[jimmydurkin]

We do not currently create VMs in the shared VPC from the service projects. Our gke host project(which does have network roles) provides the compute for our workloads

[rileykarson]

@jimmydurkin, just to clarify your environment once again: the host and service project are already established, and the shared network is already set up on both the host and service side. And your service account has no roles at all on the host project?

From your config above, projects/gke-host-project is a third, separate, project?

Sorry for the back and forth! I'd assumed networkUser (which grants projects.get) was required on the host, but https://cloud.google.com/sql/docs/mysql/private-ip#api_and_iam_requirements indicates that it isn't.

edit: Ah, never mind, I seem to have gotten it!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!