Add retries/backoff when (only) reading IAM policies

[emilymye]

We currently do a lot of retry logic around modifying IAM policies, but most resources call the IAM policy read command without retry/backoff logic. This is (potentially) leading to issues like b/130308440 where multiple members being added at the same time is hitting quotas during just the read.

---

[all]

[terraform]

Add retries/backoff when (only) reading IAM policies

[terraform-beta]

Add retries/backoff when (only) reading IAM policies

[ansible]

[inspec]

[modular-magician]

Hi! I'm the modular magician, I work on Magic Modules.
This PR seems not to have generated downstream PRs before, as of 24037f9.

Pull request statuses

No diff detected in terraform-google-conversion.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I built this PR into one or more new PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google-beta#620
depends: hashicorp/terraform-provider-google#3455

[rileykarson]

This seems similar to https://github.com/GoogleCloudPlatform/magic-modules/pull/1609/files; what do you think about using https://github.com/terraform-providers/terraform-provider-google/blob/master/google/utils.go#L337 or similar instead? Our API technically prefers exponential backoff, but standardising our retry logic feels worth not listening.

[emilymye]

@rileykarson Originally I wasn't sure if this was related to an issue with retryTime but it eventually morphed into something that was retryTime. I'll change it.

[modular-magician]

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, f1bf52b.

Pull request statuses

terraform-provider-google-beta already has an open PR.
No diff detected in terraform-google-conversion.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

[rileykarson]

LGTM! Not sure if the test is necessary now that we're using a standard retry fn

[modular-magician]

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 08c04b8.

Pull request statuses

terraform-provider-google-beta already has an open PR.
No diff detected in terraform-google-conversion.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.