add iam workload identity federation #4071

wvanderdeijl · 2020-10-08T14:35:28Z

This is my attempt to add the new Google Workload Identity Federation resources.

fixes hashicorp/terraform-provider-google#7455

But I run into the issue that the IAM resources seem to be handwritten. Now that I add products/iam/terraform.yaml all the resources in that file get generated to terraform, including things like service_account and role.
What would be the best way to leave the handwritten iam resources alone, but still be able to add the new iam-beta resources for workload identity federation?

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

`google_iam_workload_identity_pool`

modular-magician · 2020-10-08T14:36:51Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@rileykarson, please review this PR or find an appropriate assignee.

products/iam/api.yaml

rileykarson · 2020-10-08T16:27:17Z

products/iam/api.yaml

+    min_version: beta
+    base_url: projects/{{project}}/locations/global/workloadIdentityPools
+    create_url: projects/{{project}}/locations/global/workloadIdentityPools?workloadIdentityPoolId={{id}}


The {{id}} field will also need to appear as a field under properties (or parameters, but being in that list instead doesn't actually do anything).

You can then add url_param_only: true to indicate that it's only used in the URL. See

magic-modules/products/servicedirectory/api.yaml

Lines 50 to 57 in 6c2be6a

- !ruby/object:Api::Type::String

name: 'location'

description: |

The location for the Namespace.

A full list of valid locations can be found by running

`gcloud beta service-directory locations list`.

required: true

url_param_only: true

for an example.

I'd also recommend avoiding the id name, just because Terraform treats id as a reserved field name. workloadIdentityPoolId from https://cloud.google.com/iam/docs/reference/rest/v1beta/projects.locations.workloadIdentityPools/create seems appropriate.

rileykarson · 2020-10-08T16:30:08Z

products/iam/api.yaml

@@ -20,6 +20,9 @@ versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: https://iam.googleapis.com/v1/
+  - !ruby/object:Api::Product::Version


Interesting- Magic Modules assumes that subsequent APIs are a rough superset of each other, but this v1beta API doesn't contain the other handful of IAM resources at all. We may need to represent this as a separate entry in the products/ folder, and override it in terraform.yaml to name it correctly.

You don't need that yet- I want to see what changes the generator makes, exactly.

Ah, yep, we'll want to add this as a new products/ entry. MM assumes that each version in an API is a rough superset of more stable versions (ga < beta < alpha < eap) because it uses a single API endpoint for all resources under a product. IAM exposes some beta-only resources at a beta endpoint but doesn't also have the GA resources there.

You'll want to create a new products/ folder- probably iambeta- and create an api.yaml and terraform.yaml there. The name (line 21 in this file) should also likely be IAMBeta. Inside terraform.yaml, you can override the default resource name using https://github.com/GoogleCloudPlatform/magic-modules/blob/master/overrides/terraform/resource_override.rb#L34-L38 so that the resources are google_iam_* not google_iam_beta_*

This should result in a second IAM basepath/endpoint getting registered, which points to the beta endpoint.

rileykarson · 2020-10-08T16:30:57Z

products/iam/api.yaml

+          access again.
+  - !ruby/object:Api::Resource
+    name: 'WorkloadIdentityPoolProvider'


Would you mind breaking this resource into a followup PR afterwards? In my experience it tends to be a lot easier if we're iterating on one resource at a time, especially because a lot of comments that apply to one resource will apply to the second.

I've removed WorkloadIdentityPoolProvider for now and we can add that in a subsequent PR

modular-magician · 2020-10-09T13:49:27Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 68 insertions(+), 16 deletions(-))
Terraform Beta: Diff ( 9 files changed, 739 insertions(+), 18 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 13 deletions(-))
Inspec: Diff ( 16 files changed, 80 insertions(+), 16 deletions(-))

wvanderdeijl · 2020-10-09T14:01:29Z

I added a first attempt at a test, but can't figure out why this is failing when running the terraform-provider-google-beta tests:

terraform-provider-google-beta % make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMWorkloadIdentityPool'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMWorkloadIdentityPool -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMWorkloadIdentityPool_example
=== PAUSE TestAccIAMWorkloadIdentityPool_example
=== CONT  TestAccIAMWorkloadIdentityPool_example
2020/10/09 15:40:12 [WARN] Got error running Terraform: 2020/10/09 15:40:11 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0

Error: Invalid resource type

  on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest228702188/work766105179/config285216703/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
  13: resource "google_iam_workload_identity_pool" "my_pool" {

The provider provider.google does not support resource type
"google_iam_workload_identity_pool".

    TestAccIAMWorkloadIdentityPool_example: provider_test.go:266: Step 1/2 error: Error running pre-apply refresh: 2020/10/09 15:40:11 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest228702188/work766105179/config285216703/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".
        
--- FAIL: TestAccIAMWorkloadIdentityPool_example (4.01s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 5.022s
FAIL
make: *** [testacc] Error 1

This is weird as I can see that google_iam_workload_identity_pool is generated in the list of resources in provider.go

modular-magician · 2020-10-09T14:16:06Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 75 insertions(+), 23 deletions(-))
Terraform Beta: Diff ( 9 files changed, 746 insertions(+), 25 deletions(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 13 deletions(-))
Inspec: Diff ( 16 files changed, 80 insertions(+), 16 deletions(-))

rileykarson

Sorry for the delay in getting back!

The error you're getting is because the tests are being run in the GA provider and not the google-beta provider. The GA google provider doesn't support beta features.

You'll want to supply -v beta to MM to generate into the terraform-google-provider-beta repo.

third_party/terraform/tests/resource_iam_workload_identity_pool_test.go

products/iam/api.yaml

rileykarson · 2020-10-13T16:39:05Z

products/iam/api.yaml

@@ -20,6 +20,9 @@ versions:
  - !ruby/object:Api::Product::Version
    name: ga
    base_url: https://iam.googleapis.com/v1/
+  - !ruby/object:Api::Product::Version


Ah, yep, we'll want to add this as a new products/ entry. MM assumes that each version in an API is a rough superset of more stable versions (ga < beta < alpha < eap) because it uses a single API endpoint for all resources under a product. IAM exposes some beta-only resources at a beta endpoint but doesn't also have the GA resources there.

You'll want to create a new products/ folder- probably iambeta- and create an api.yaml and terraform.yaml there. The name (line 21 in this file) should also likely be IAMBeta. Inside terraform.yaml, you can override the default resource name using https://github.com/GoogleCloudPlatform/magic-modules/blob/master/overrides/terraform/resource_override.rb#L34-L38 so that the resources are google_iam_* not google_iam_beta_*

This should result in a second IAM basepath/endpoint getting registered, which points to the beta endpoint.

wvanderdeijl · 2020-10-13T19:56:29Z

I've moved the new resources to products/iambeta/api.yaml and running bundle exec compiler -v "beta" -e terraform -o "$GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta" -a seems to generate okay source code for terraform. This also includes my (beta) tests, but unfortunately I still cannot run the test (not sure why):

make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMBetaWorkloadIdentityPool_example'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMBetaWorkloadIdentityPool_example -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/13 21:52:00 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest637359650/work048036121/config035092829/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".

But I can see the new google_iam_workload_identity_pool resource is being generated into google-beta/provider.go so not really sure what is going there

Add the google-beta provider to the example also doesn't work:

resource "google_iam_workload_identity_pool" "my_pool" {
  provider     = google-beta

gives

=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/13 21:57:50 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Could not satisfy plugin requirements
        
        
        Plugin reinitialization required. Please run "terraform init".
        
        Plugins are external binaries that Terraform uses to access and manipulate
        resources. The configuration provided requires plugins which can't be located,
        don't satisfy the version constraints, or are otherwise incompatible.
        
        Terraform automatically discovers provider requirements from your
        configuration, including providers used in child modules. To see the
        requirements and constraints from each module, run "terraform providers".
        
        
        
        Error: provider.google-beta: no suitable version installed
          version requirements: "(any version)"
          versions installed: none

Is there another beta-only resource with a test I can copy from?

modular-magician · 2020-10-13T19:57:21Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 127 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 8 files changed, 698 insertions(+), 3 deletions(-))

modular-magician · 2020-10-13T19:58:26Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

modular-magician · 2020-10-13T19:59:45Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

wvanderdeijl · 2020-10-13T20:01:18Z

Most changes to identity-pools (and providers) return Operations that we need to poll for. I can copy that from other similar resources, but would like to get the test running before attempting so. With a (failing) test it's much easier for me to iterate on attempts to get it all working.

rileykarson

Are you still encountering the same issue with the branch as it is now? I received the following, which indicates the rest ran successfully:

------- Stdout: -------
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/14 21:29:32 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0
Error: Missing required argument
on config516010655/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
13: resource "google_iam_workload_identity_pool" "my_pool" {
The argument "workload_identity_pool_id" is required, but no definition was
found.
--- FAIL: TestAccIAMBetaWorkloadIdentityPool_example (0.39s)
FAIL

One thing you could try is checking out https://github.com/modular-magician/terraform-provider-google-beta/tree/auto-pr-4071, the code generated by our CI by your current changes (and the code I ran), and then running MM on top of that locally.

products/iambeta/api.yaml

wvanderdeijl · 2020-10-16T07:40:41Z

I must be doing something wrong with starting the tests. I've checkout your branch https://github.com/modular-magician/terraform-provider-google-beta/tree/auto-pr-4071 and tried to run the tests and run into the same errors:

% make testacc TEST=./google-beta TESTARGS='-run=TestAccIAMBetaWorkloadIdentityPool_example'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccIAMBetaWorkloadIdentityPool_example -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccIAMBetaWorkloadIdentityPool_example
=== PAUSE TestAccIAMBetaWorkloadIdentityPool_example
=== CONT  TestAccIAMBetaWorkloadIdentityPool_example
    TestAccIAMBetaWorkloadIdentityPool_example: provider_test.go:266: Step 1/1 error: Error running pre-apply refresh: 2020/10/15 15:07:25 [DEBUG] Using modified User-Agent: Terraform/0.12.24 HashiCorp-terraform-exec/0.10.0
        
        Error: Invalid resource type
        
          on ../../../../../../../../var/folders/wf/xm7lxbtn1nbbvzfzb3r4v89r0000gn/T/tftest422138618/work352437777/config639766485/terraform_plugin_test.tf line 13, in resource "google_iam_workload_identity_pool" "my_pool":
          13: resource "google_iam_workload_identity_pool" "my_pool" {
        
        The provider provider.google does not support resource type
        "google_iam_workload_identity_pool".
        
--- FAIL: TestAccIAMBetaWorkloadIdentityPool_example (3.13s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 3.803s
FAIL
make: *** [testacc] Error 1

~~This is using the test command I got from the contribution guide at https://github.com/hashicorp/terraform-provider-google/blob/master/.github/CONTRIBUTING.md:~~

make testacc TEST=./google TESTARGS='-run=TestAccContainerNodePool_basic'

~~The only thing I changed is the TEST argument to ./google-beta since I am testing the beta provider. Is there something I'm missing?~~

Nevermind, I forgot to upgrade terraform to the latest version. Now upgrade to v0.13.4 and the test runs (and fails as expected). While I continu to work on this PR I'll mark it as draft until it is complete

so we first focus on a single new resource

as this is no auto-generated

modular-magician · 2020-10-16T07:51:40Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 125 insertions(+))
Terraform Beta: Diff ( 7 files changed, 696 insertions(+), 2 deletions(-))

modular-magician · 2020-10-16T16:07:08Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 165 insertions(+))
Terraform Beta: Diff ( 10 files changed, 1047 insertions(+), 2 deletions(-))
TF OiCS: Diff ( 8 files changed, 213 insertions(+))

wvanderdeijl · 2020-10-16T16:24:13Z

I have added some tests and made some changes to the code to ensure all tests succeed. I think this is ready for review by @rileykarson

wvanderdeijl · 2020-10-17T06:08:22Z

Should I also make sure a terraform data source is generated? (if so, what are the basic steps to get a data source - are they all handcoded or somehow generated?)

rileykarson

LGTM!

If you're interested in adding a datasource, #4111 is a good template for what adding one looks like. They're handwritten, but based off the existing resource code. I'd suggest doing that in a separate PR (including from the one adding WorkloadIdentityPoolProvider) for ease of review.

google-cla bot added the cla: yes label Oct 8, 2020

wvanderdeijl force-pushed the workload-identity-federration branch from be9866b to 05402fa Compare October 8, 2020 14:36

modular-magician requested a review from rileykarson October 8, 2020 14:36

rileykarson reviewed Oct 8, 2020

View reviewed changes

wvanderdeijl force-pushed the workload-identity-federration branch from 05402fa to 7e4e4ba Compare October 9, 2020 13:41

wvanderdeijl requested a review from rileykarson October 9, 2020 14:14

rileykarson reviewed Oct 13, 2020

View reviewed changes

wvanderdeijl force-pushed the workload-identity-federration branch 2 times, most recently from b09bf11 to 698a885 Compare October 13, 2020 19:51

wvanderdeijl requested a review from rileykarson October 13, 2020 20:01

rileykarson reviewed Oct 14, 2020

View reviewed changes

products/iambeta/api.yaml Outdated Show resolved Hide resolved

wvanderdeijl added 8 commits October 16, 2020 09:41

add workload identity federation

c89e836

remove WorkloadIdentityPoolProvider for now

8ce2623

so we first focus on a single new resource

exclude WorkloadIdentityPool from ansible and inspec

e0b0720

do not use reserved property id

c2ad00e

do not generate handwritten iam resources

cb8e04c

remove custom endpoint for handwritten iam resources

5b7de59

as this is no auto-generated

example test

071d927

also mention non-beta URL in docs

771269d

wvanderdeijl added 2 commits October 16, 2020 09:41

split iam and iambeta product

e73a919

wvanderdeijl force-pushed the workload-identity-federration branch from 698a885 to 9e09387 Compare October 16, 2020 07:42

wvanderdeijl marked this pull request as draft October 16, 2020 08:01

add tests and async operations

1754b9a

wvanderdeijl marked this pull request as ready for review October 16, 2020 16:23

wvanderdeijl requested a review from rileykarson October 16, 2020 16:24

rileykarson approved these changes Oct 20, 2020

View reviewed changes

rileykarson merged commit 18cff41 into GoogleCloudPlatform:master Oct 20, 2020

This was referenced Oct 20, 2020

add iam workload identity federation hashicorp/terraform-provider-google-beta#2623

Merged

add iam workload identity federation hashicorp/terraform-provider-google#7580

Merged

This was referenced Oct 20, 2020

add iam workload identity pool provider #4129

Merged

adding data source for identity pool #4134

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add iam workload identity federation #4071

add iam workload identity federation #4071

wvanderdeijl commented Oct 8, 2020 •

edited

Loading

modular-magician commented Oct 8, 2020

rileykarson Oct 8, 2020

rileykarson Oct 8, 2020

rileykarson Oct 13, 2020

rileykarson Oct 8, 2020

wvanderdeijl Oct 9, 2020

modular-magician commented Oct 9, 2020

wvanderdeijl commented Oct 9, 2020

modular-magician commented Oct 9, 2020

rileykarson left a comment

rileykarson Oct 13, 2020

wvanderdeijl commented Oct 13, 2020 •

edited

Loading

modular-magician commented Oct 13, 2020

modular-magician commented Oct 13, 2020

modular-magician commented Oct 13, 2020

wvanderdeijl commented Oct 13, 2020

rileykarson left a comment

wvanderdeijl commented Oct 16, 2020 •

edited

Loading

modular-magician commented Oct 16, 2020

modular-magician commented Oct 16, 2020

wvanderdeijl commented Oct 16, 2020

wvanderdeijl commented Oct 17, 2020 •

edited

Loading

rileykarson left a comment

	- !ruby/object:Api::Type::String
	name: 'location'
	description: \|
	The location for the Namespace.
	A full list of valid locations can be found by running
	`gcloud beta service-directory locations list`.
	required: true
	url_param_only: true

add iam workload identity federation #4071

add iam workload identity federation #4071

Conversation

wvanderdeijl commented Oct 8, 2020 • edited Loading

modular-magician commented Oct 8, 2020

rileykarson Oct 8, 2020

Choose a reason for hiding this comment

rileykarson Oct 8, 2020

Choose a reason for hiding this comment

rileykarson Oct 13, 2020

Choose a reason for hiding this comment

rileykarson Oct 8, 2020

Choose a reason for hiding this comment

wvanderdeijl Oct 9, 2020

Choose a reason for hiding this comment

modular-magician commented Oct 9, 2020

Diff report:

wvanderdeijl commented Oct 9, 2020

modular-magician commented Oct 9, 2020

Diff report:

rileykarson left a comment

Choose a reason for hiding this comment

rileykarson Oct 13, 2020

Choose a reason for hiding this comment

wvanderdeijl commented Oct 13, 2020 • edited Loading

modular-magician commented Oct 13, 2020

Diff report:

modular-magician commented Oct 13, 2020

Diff report:

modular-magician commented Oct 13, 2020

Diff report:

wvanderdeijl commented Oct 13, 2020

rileykarson left a comment

Choose a reason for hiding this comment

wvanderdeijl commented Oct 16, 2020 • edited Loading

modular-magician commented Oct 16, 2020

Diff report:

modular-magician commented Oct 16, 2020

Diff report:

wvanderdeijl commented Oct 16, 2020

wvanderdeijl commented Oct 17, 2020 • edited Loading

rileykarson left a comment

Choose a reason for hiding this comment

wvanderdeijl commented Oct 8, 2020 •

edited

Loading

wvanderdeijl commented Oct 13, 2020 •

edited

Loading

wvanderdeijl commented Oct 16, 2020 •

edited

Loading

wvanderdeijl commented Oct 17, 2020 •

edited

Loading