IAM issues due to deleted IAM member

[slevenick]

Community Note

This is a canonical bug for IAM issues related to the deleted: IAM member syntax that is being rolled out soon. I expect to see bug reports due to this new behavior, this bug should contain information on how to handle the transition period while this is being rolled out.

Please check out the IAM deleted principals guide to see if it describes the problem you are seeing. If not, but the issue seems to be related to deleted principals, please comment with a description of your issue. If the issue does not seem to be related to deleted principals, please file a separate issue with the relevant information.

Terraform Version

All versions. This is due to an API-level change.

If you are seeing badRequest errors and are using a version lower than 2.20.1 you will need to upgrade. Versions below 2.20.1 and 3.3.0 had a bug related to deleted serviceAccounts that would cause a badRequest error. See: https://github.com/hashicorp/terraform-provider-google/blob/master/CHANGELOG.md#330-december-17-2019

Affected Resource(s)

• *_iam_policy
• *_iam_binding
• *_iam_member

Expected Failures

Several different failure modes are expected. The main issue will be permadiffs that are not resolved during apply due to the API interpreting a non-deleted IAM member as a reference to a deleted IAM member with the same email on the same policy. Another issue that may arise will be API-level restrictions rejecting IAM policies for containing references to both a deleted: and non-deleted member in the same policy.

Finally, iam_member resources may fail with Provider produced inconsistent result after apply if the IAM policy has a reference to the deleted form of the same member. These errors should go away as the rollout completes and be replaced by an API-level rejection of the policy.

More information can be found here: https://www.terraform.io/docs/providers/google/guides/iam_deleted_members.html

References

• Question regarding the upcoming IAM behaviour change on 27th July #6736
• Handle deleted: prefix when deduplicating IAM member map GoogleCloudPlatform/magic-modules#2819
• Official docs on deleted members in policies

[ct-dh]

We see the scenario of deleted members remaining in policies fairly often in our dev/testing environments because we have multiple teams working that have shared ownership of certain resources between different deployment configurations. Things like deployment failures and failed cleanups is usually where this type of thing occurs.

In case it is useful for anyone else, in some cases we use the below to clean up stale bindings in pipelines if they are expected to occur, prior to running terraform:

Project level IAM bindings:

PROJECT_POLICY="/tmp/iam-policy-${PROJECT}"
gcloud projects get-iam-policy "${PROJECT}" --format="yaml" > "${PROJECT_POLICY}-original"
cat "${PROJECT_POLICY}-original" | grep -Ev '^\s+\- deleted:' > "${PROJECT_POLICY}-edited"
gcloud projects set-iam-policy "${PROJECT}" "${PROJECT_POLICY}-edited" > "${PROJECT_POLICY}-updated"

Bucket bindings:

for bucket in $(gsutil ls -p $GCP_PROJECT);
do
        for user in $(gsutil iam get $bucket | grep -E '^\s+"deleted:serviceAccount' | sed 's/"//g' | sed 's/,//' | sort | uniq); do
                gsutil iam ch -d "${user}" "${bucket}"
        done
done

[slevenick]

I thought that we would see issues related to this, but it appears that everything is working out reasonably well. Closing this issue for now as the rollout should be complete and no major issues were reported.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!