google_project_iam_member/google_project_iam_binding Fails for roles/cloudsql.client, Works for Other #5107
Comments
|
I've been noticing the same error across many different projects as of today: For example, this config is causing this error: The error is quite confusing, because I think the right fix is likely to filter out deleted principles when sending the IAM policy back. |
|
I've been doing a bit more investigation into this (tracked in #333). I've been able to consistently reproduce it on my project, here are the debug logs. Looking at the logs, I suspect the issue is related to deleted IAM principles. Specifically, I see that we attempt to reflect a deleted IAM principle back in the setPolicy response. I've also done some version testing:
Right now the best workaround I can find is to pin the provider to |
|
I've got a fix for this on the way: GoogleCloudPlatform/magic-modules#2819 |
|
As a workaround until the fix is released you can delete service account IAM members with the This issue is caused specifically by deleted service accounts that exist on the resource that terraform is managing members on, so removing references to them will allow terraform to work normally. |
|
This fix is available now in the |
|
@slevenick I've just attempted it after pinning
|
|
@jjorissen52 can you provide debug logs for the failing run? That will help me debug what is going on |
|
@slevenick unfortunately, earlier today I bumped up to |
|
The 3.3.0 release is expected to go out tomorrow which has this fix. Please let me know if you encounter the same issue with that version, but I'll close this until then. I believe this issue has been fixed with 2.20.1 as I am unable to reproduce issues at this point Downgrading from 3.x to 2.x is going to be difficult and not recommended |
|
I am definitely still encountering this issue with I also upgraded everything to |
|
I have just tried this with version 3.4.0 and I am getting the same error, here's a code snippet: Error output: |
|
@madmaze or @lobsterdore can you include a debug log for the failed apply? I am able to apply the config provided with |
|
@slevenick , I just upgraded to v3.4.0 and can confirm that this is still affecting me. Debug Logs
Shows same error as before: |
|
@jjorissen52 That is odd. Can you apply the same config on a new (clean) project? I suspect that there is something strange happening with the IAM policy for your existing project. I believe this is an unrelated issue, but it presents with the same (not very helpful) error message. Looking at the debug log, I would guess that this is causing the failure: Terraform receives an IAM policy that has a series of members named I believe that removing these faulty members will cause terraform to succeed. Could you try either using the console or gcloud to remove these members, or using a |
|
@slevenick Apologies, I manually modified those lines so as to not publish my co-workers email addresses. each of those lines once contained an |
|
Ok that makes sense. I'm back to being confused about why this is happening. This seems unrelated to the other issues around Were you able to successfully apply this config with versions of the provider after What I'm trying to figure out is if this broke with the |
|
@slevenick I had never attempted this particular role assignment ( Unfortunately, I cannot tell if this is the version that was used when creating the binding or if I've since updated the version; the state history does not seem to contain information about provider versions. |
|
I have a debug log of both I've cleaned up two snippets, 2.12.0 & 2.20.1 which seem relevant to me. Looks like besides the order, the sent data is exactly the same besides the Two other differences seem to be in the headers:
|
|
I am also seeing this issue when applying iam_member with
In the debug logs, I am seeing this: |
|
@michyliao that looks like a different issue. Can you file a separate issue with debug logs included? |
|
I'm unable to track this down by just the error message from the debug logs (invalid argument is very generic) I'll probably need to be able to reproduce this to make further progress. @madmaze can you send me the full debug logs for a failing run? It would help to have the full request/response pair without any changes. If you don't want to post them publicly could you send them to my username @google.com |
|
@slevenick It seems that, for the affected project, |
|
Just today faced this bug and am very surprised that it's not fixed for months. The error message " Error 400: Request contains an invalid argument., badReques" is misleading. As I wrote above the actual error is Capital letters in project user ID (actually in our case with "owner" permissions if that makes any change) What's the most weird in this situation is that I can't add that user back with low case letters. Google checks the email I provide (lower case) in its user database(s) and adds it with Capital letters again. Please fix. |
|
Hey @akrasnov-drv sorry that this caused issues for you. How are you adding back the user with lower case letters? Can you give me an overview of your workflow, like are you using terraform to attempt to add this user back, but it gets sent as |
|
Hi @slevenick |
|
Hm, can you provide debug logs for the failing run? I'm unable to create a user with capital letters in their name. I have created a user with capital letters, but the IAM console only finds it as lowercase, which doesn't cause any issues. |
|
Yes, sure. Test code account_id = "del-me"
display_name = "bug test sa"
}
resource "google_project_iam_member" "bug_test_role" {
role = "roles/compute.instanceAdmin"
member = "serviceAccount:${google_service_account.del_me.email}"
depends_on = [google_service_account.del_me]
}Error The log (attached, with some security related masking) is for google-beta but it fails the same way for google too. |
|
That's very unusual. How did you create the user with capital letters, is it just an old email that existed? And you have found that removing the user with capital letters allows you to apply the binding? I'll ask around for why the API would be returning upper case values and if this is intended we should handle this correctly in Terraform |
There are enough complaints in Internet regarding these functions not working. I believe all (or most) of them have this issue (user(s) with Upper case letter(s)). This should be handled by terraform provider. I do not believe Google will update it user databases (or API)... |
|
@jjorissen52 does your IAM policy have users with upper case letters? I'm tracking down the intended behavior here, and will definitely handle this in the provider if needed |
|
@slevenick The project does have one user with capital letters in the email, though none of bindings defined via terraform do anything with that user. Don't know if that makes a difference. |
|
Yes, I also do nothing with the problem user. But you can see it in debug and it brakes the workflow (I mean just existence of it). |
|
@akrasnov-drv @slevenick That was it.
|
|
@akrasnov-drv thank you for figuring out the root cause of this issue! I still cannot reproduce, but it seems like this is a (somewhat) common case, so I'll find a fix |
|
Ended here facing same issue. I was using fiexed using: serviceAccount:[email protected] It's in doc anyway. |
|
I'm still having trouble reproducing this issue, and I believe that there is something strange going on with the particular emails being used here as emails are not handled case sensitively by the API. Can I have one of you @akrasnov-drv or @jjorissen52 send me the actual email that is causing the problems? It will help me track down what exactly about these users is causing the issue. You can send it to my github username @google.com |
|
Hi, |
slevenick
added
the
persistent-bug
|
I've hit the same issue today running terraform gke public module I believe that the issue happens when attempting to add a role to a new service account (existing policy), you have to first fetch the policy which includes the user with the capital letter, then append to it and apply it. If you can point me to the code where this is done I can try to replicate it using gcloud CLI, and see if its an SKD issue or implementation issue (usually the SDK will make fixes to it before applying it) update: Im unable to replicate it on a single role, already containing a CamelCase user name, maybe its an issue with size of the payload? resource "google_service_account" "sa" {
account_id = "terratest"
display_name = "Terratest Service Account"
}
resource "google_project_iam_member" "log_writer" {
project = "ami-playground"
role = "roles/logging.logWriter"
member = "serviceAccount:${google_service_account.sa.email}"
} |
|
Surprisingly I'm unable to reproduce this issue in my own project. If I add a user with a capital letter, it behaves the same way as in all of the cases described here, where Terraform lowercases any capital letters coming from the API, but in all of my cases the API accepts the lowercase version. For example, the API will return: I add a binding with a different user, posting back a policy with Which the API accepts and automatically corrects and returns I'm trying to debug with the team internally, and may reach out to some of you for help in reproducing this for them |
|
@slevenick |
|
I was just experiencing what seems like a related issue to this and #4276 and was able to solve it. Maybe this can help others in the thread. I have a Returned the resource "google_project_iam_member" "mem" {
role = "roles/${google_project_iam_custom_role.role.role_id}"
member = "serviceAccount:${data.google_service_account.sa.email}"
}Succeeded: resource "google_project_iam_member" "mem" {
role = "projects/${var.project}/roles/${google_project_iam_custom_role.role.role_id}"
member = "serviceAccount:${data.google_service_account.sa.email}"
} |
|
Yes, #4276 is related, and @danawillow has a working reproduction of this issue, so hopefully we should get it fixed soon! I'll close this as a duplicate at this point as #4276 is the same issue |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks! |
ghost
locked and limited conversation to collaborators
Community Note
Terraform Version
Affected Resource(s)
Terraform Configuration Files
Debug Output
https://gist.github.com/jjorissen52/d253d274cdb763b47b55cbe3ee0f19e2
Expected Behavior
Binding should happen
Actual Behavior
Steps to Reproduce
terraform applyImportant Factoids
I have been able to use this exact resource setup to apply other roles to other service accounts.
References
Resolution here does not seem to work:
The text was updated successfully, but these errors were encountered: