google_project_iam_member Policy update access denied

[Leo843]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.7                                                                                                                                                                       
on linux_amd64                                                                                                                                                                         
+ provider registry.terraform.io/hashicorp/google v3.84.0                                                                                                                              
+ provider registry.terraform.io/hashicorp/google-beta v3.84.0

Affected Resource(s)

• google_project_iam_member
• google_project_iam_binding (possibly)

Terraform Configuration Files

provider "google" {                                                                                                                                                                    
  project = var.project                                                                                                                                                                
  region  = var.region                                                                                                                                                                 
}                                                                                                                                                                                      
                                                                                                                                                                                       
resource "google_service_account" "service_account" {                                                                                                                                  
  project      = var.project                                                                                                                                                           
  account_id   = var.name                                                                                                                                                              
  display_name = var.description                                                                                                                                                       
}                                                                                                                                                                                      
                                                                                                                                                                                       
resource "google_project_iam_member" "service_account-roles" {                                                                                                                         
  project = var.project                                                                                                                                                                
  role    = "roles/logging.logWriter"                                                                                                                                                  
  member  = "serviceAccount:${google_service_account.service_account.email}"                                                                                                           
}

Debug Output

https://gist.github.com/Leo843/77cc7e919b80958ff911d9b979b9bc5e

Expected Behavior

The role is given to the service account.

Actual Behavior

Google API returns 403 Policy update access denied when trying to set the role to the service account.

Steps to Reproduce

• Activate a service account gcloud auth activate-service-account --key-file=<key-file>
• Make sure that the service account is allowed to add/remove IAM policies

  gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/monitoring.viewer
  

  gcloud projects remove-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/monitoring.viewer
  

• Provide valid input variables for the configuration given above
• Run terraform apply

Important Factoids

terraform apply is executed with a service account used for deployments. The service account has necessary permissions to grant roles (running gcloud projects add/remove-iam-policy-binding succeed).

[edwardmedia]

@Leo843 do you have environment GOOGLE_CLOUD_KEYFILE_JSON or other environment variables set on the Terraform runner? Can you share the debug logs for about two gcloud executions?

[Leo843]

There is no GOOGLE_* environment variables set.

Here is the output for gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/logging.logWriter --verbosity=debug. The debug output is rather minimal. I don't know if there is a way to get more than the --verbosity=debug flag.

DEBUG: Running [gcloud.projects.add-iam-policy-binding] with arguments: [--member: "serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com", --role: "roles/logging.logWriter", --verbosity: "debug", PROJECT_ID: "<PROJECT_ID>"]
DEBUG: Starting new HTTPS connection (1): cloudresourcemanager.googleapis.com:443
DEBUG: https://cloudresourcemanager.googleapis.com:443 "POST /v1/projects/<PROJECT_ID>:getIamPolicy?alt=json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): cloudresourcemanager.googleapis.com:443
DEBUG: https://cloudresourcemanager.googleapis.com:443 "POST /v1/projects/<PROJECT_ID>:setIamPolicy?alt=json HTTP/1.1" 200 None
Updated IAM policy for project [<PROJECT_ID>].
INFO: Display format: "default"
bindings:
...
- members:
  - serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com
  role: roles/logging.logWriter
...
etag: BwXMfSnscv0=
version: 1

Here is the output for gcloud projects remove-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/logging.logWriter --verbosity=debug.

DEBUG: Running [gcloud.projects.remove-iam-policy-binding] with arguments: [--member: "serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com", --role: "roles/logging.logWriter", --verbosity: "debug", PROJECT_ID: "<PROJECT_ID>"]
DEBUG: Starting new HTTPS connection (1): cloudresourcemanager.googleapis.com:443
DEBUG: https://cloudresourcemanager.googleapis.com:443 "POST /v1/projects/<PROJECT_ID>:getIamPolicy?alt=json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): cloudresourcemanager.googleapis.com:443
DEBUG: https://cloudresourcemanager.googleapis.com:443 "POST /v1/projects/<PROJECT_ID>:setIamPolicy?alt=json HTTP/1.1" 200 None
Updated IAM policy for project [<PROJECT_ID>].
INFO: Display format: "default"
bindings:
...
etag: BwXMfXEg6-Q=
version: 1

[edwardmedia]

@Leo843 can you try adding --log-http in your gcloud command?

[Leo843]

Oh i did not know about this flag, it could have save time, sorry.

gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/logging.logWriter --verbosity=debug --log-http
https://gist.github.com/Leo843/578286d57eb1cabaf6bb581cf27453d2

gcloud projects remove-iam-policy-binding <PROJECT_ID> --member=serviceAccount:test-iam-sa@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/logging.logWriter --verbosity=debug --log-http
https://gist.github.com/Leo843/d0312a9506b28aa45e728f5893779ace

[Leo843]

I tried using a user (with roles/owner) instead of a service account and i get the same errors.

[Leo843]

Ok, I found the error.

It did not work because GOOGLE_APPLICATION_CREDENTIALS was not set (I thought that login in with gcloud set the env automatically, but this is not the case). However, I don't understand why the provider was able the create a service account (even though setting the IAM fails) without credentials.

So this is not a bug but a misunderstanding.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.