Composer v2 (beta) (#9611) #5181

kubasieron · 2021-09-08T15:07:27Z

Add support for creation of Cloud Composer v2 environments.

fixes hashicorp/terraform-provider-google/issues/9611

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

composer: added support for composer v2 fields `workloads_config` and `cloud_composer_network_ipv4_cidr_block` to `composer_environment`

modular-magician · 2021-09-08T15:08:51Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@ScottSuarez, please review this PR or find an appropriate assignee.

modular-magician · 2021-09-08T15:18:54Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 3 files changed, 305 insertions(+), 1 deletion(-))

ScottSuarez · 2021-09-10T18:01:47Z

/gcbrun

modular-magician · 2021-09-10T18:13:27Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 3 files changed, 305 insertions(+), 2 deletions(-))

ScottSuarez · 2021-09-17T00:02:43Z

might want to rebase to unblock this

/gcbrun

modular-magician · 2021-09-17T00:12:33Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 2 files changed, 304 insertions(+))

modular-magician · 2021-09-20T09:14:02Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 3 files changed, 305 insertions(+), 1 deletion(-))

modular-magician · 2021-09-20T10:59:50Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccComposerEnvironment_ComposerV2|TestAccComputeManagedSslCertificate_managedSslCertificateBasicExample|TestAccComputeServiceAttachment_serviceAttachmentBasicExample You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=206579

modular-magician · 2021-09-20T11:09:30Z

Tests failed during RECORDING mode: TestAccComposerEnvironment_ComposerV2|TestAccComputeServiceAttachment_serviceAttachmentBasicExample Please fix these to complete your PR

ScottSuarez · 2021-09-20T18:29:17Z

/gcbrun

modular-magician · 2021-09-20T18:41:06Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 4 files changed, 306 insertions(+), 3 deletions(-))

modular-magician · 2021-09-20T20:26:42Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccComposerEnvironment_ComposerV2|TestAccComputeForwardingRule_update|TestAccComputeServiceAttachment_serviceAttachmentBasicExample You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=206608

modular-magician · 2021-09-20T20:38:48Z

Tests failed during RECORDING mode: TestAccComposerEnvironment_ComposerV2|TestAccComputeServiceAttachment_serviceAttachmentBasicExample Please fix these to complete your PR

ScottSuarez · 2021-09-21T00:44:13Z

I keep getting SA issues

Error: googleapi: Error 400: Composer API Service Agent service account does not have required permissions set. Please refer to https://cloud.google.com/composer/docs/how-to/managing/creating and Composer Creation Troubleshooting pages to resolve this issue., failedPrecondition

I the sa that runs the tests permissions. Do I have to grant another service account permissions?

ScottSuarez

Can you please also add these fields to the documentation for this resource

https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/website/docs/r/composer_environment.html.markdown

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb

modular-magician · 2021-09-21T08:09:10Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 2 files changed, 307 insertions(+))

modular-magician · 2021-09-21T09:56:25Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccComposerEnvironment_ComposerV2|TestAccComputeServiceAttachment_serviceAttachmentBasicExample|TestAccOrgPolicyPolicy_EnforcePolicy|TestAccOrgPolicyPolicy_FolderPolicy|TestAccOrgPolicyPolicy_OrganizationPolicy|TestAccOrgPolicyPolicy_ProjectPolicy You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=206675

modular-magician · 2021-09-21T11:00:48Z

Tests failed during RECORDING mode: TestAccComputeServiceAttachment_serviceAttachmentBasicExampleUpdate|TestAccComposerEnvironment_ComposerV2 Please fix these to complete your PR

modular-magician · 2021-09-21T13:42:18Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 7 insertions(+))
Terraform Beta: Diff ( 3 files changed, 308 insertions(+), 1 deletion(-))

modular-magician · 2021-09-21T15:29:41Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccComposerEnvironment_ComposerV2 You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=206680

modular-magician · 2021-09-21T16:38:13Z

Tests failed during RECORDING mode: TestAccComposerEnvironment_ComposerV2 Please fix these to complete your PR

ScottSuarez · 2021-09-21T17:34:38Z

The test failed with the following error

Error: Error waiting to create Environment: Error waiting for Creating Environment: Error code 9, message: Some of the GKE pods failed to become healthy. Please check the GKE logs for details, and retry the operation

ScottSuarez · 2021-09-21T17:37:48Z

/gcbrun

ScottSuarez · 2021-09-21T18:23:52Z

/gcbrun

ScottSuarez · 2021-09-21T18:30:12Z

/gcbrun

modular-magician · 2021-09-28T09:14:19Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 16 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 4 files changed, 409 insertions(+), 2 deletions(-))

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb

ScottSuarez · 2021-09-28T19:51:45Z

mmv1/third_party/terraform/website/docs/r/composer_environment.html.markdown

@@ -186,6 +186,10 @@ The `config` block supports:
 * `maintenance_window` -
  (Optional, [Beta](https://terraform.io/docs/providers/google/guides/provider_versions.html))
  The configuration settings for Cloud Composer maintenance window.
+
+* `workloads_config` -


This field is not beta correct?

can you expand workloads_config subfields inline with the other block types

ie: The workloads_config block supports:

ScottSuarez · 2021-09-28T19:53:09Z

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb

+    // version_1 - write own flatten for Scheduler, WebServer and Worker
+    // version_2 (Scott idea) - use different types for inner structs in flatten
+    // version_3 - v1 and v2 combined
+


v3 makes sense, lets go with that approach

ScottSuarez · 2021-10-04T19:25:24Z

- clean up the code for the flattener
- document the nested fields
- document which fields are composer v1 only and which are composer v2

…to be set correctly

modular-magician · 2021-10-04T19:44:04Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 32 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 3 files changed, 424 insertions(+), 3 deletions(-))

modular-magician · 2021-10-04T21:38:28Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccIapTunnelIamBindingGenerated|TestAccComposerEnvironment_update|TestAccComposerEnvironment_ComposerV2|TestAccComposerEnvironment_withDatabaseConfig|TestAccComposerEnvironment_withWebServerConfig|TestAccComputeServiceAttachment_serviceAttachmentBasicExample|TestAccComputeServiceAttachment_serviceAttachmentBasicExampleUpdate You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=208992

modular-magician · 2021-10-04T22:17:31Z

Tests failed during RECORDING mode: TestAccIapTunnelIamBindingGenerated|TestAccComposerEnvironment_ComposerV2|TestAccComposerEnvironment_update|TestAccComposerEnvironment_withDatabaseConfig|TestAccComposerEnvironment_withWebServerConfig Please fix these to complete your PR

ScottSuarez · 2021-10-04T23:37:53Z

error from api

=== RUN   TestAccComposerEnvironment_ComposerV2
=== PAUSE TestAccComposerEnvironment_ComposerV2
=== CONT  TestAccComposerEnvironment_ComposerV2
    provider_test.go:278: Step 1/3 error: Error running apply: exit status 1
        
        Error: googleapi: Error 400: Found 2 problems:
        	1) Cloud Composer Network CIDR block must have a maximum size of /24.
        	2) Cloud SQL and webserver CIDR blocks must not overlap., badRequest
        
          on terraform_plugin_test.tf line 2, in resource "google_composer_environment" "test":
           2: resource "google_composer_environment" "test" {
        
        
--- FAIL: TestAccComposerEnvironment_ComposerV2 (95.16s)

modular-magician · 2021-10-05T17:16:04Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 169 insertions(+), 32 deletions(-))
Terraform Beta: Diff ( 3 files changed, 521 insertions(+), 38 deletions(-))

modular-magician · 2021-10-05T19:06:53Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccComposerEnvironment_ComposerV2 You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=209160

modular-magician · 2021-10-05T20:36:05Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 177 insertions(+), 59 deletions(-))
Terraform Beta: Diff ( 4 files changed, 550 insertions(+), 66 deletions(-))

modular-magician · 2021-10-05T22:25:35Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccComputeManagedSslCertificate_managedSslCertificateBasicExample You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=209184

modular-magician · 2021-10-06T17:22:35Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 173 insertions(+), 59 deletions(-))
Terraform Beta: Diff ( 3 files changed, 548 insertions(+), 65 deletions(-))

modular-magician · 2021-10-06T19:16:30Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccServiceNetworkingPeeredDNSDomain_basic|TestAccPubsubLiteReservation_pubsubLiteReservationBasicExample|TestAccPubsubLiteTopic_pubsubLiteTopicBasicExample You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=209376

Co-authored-by: Scott Suarez <[email protected]> Co-authored-by: Goran Obradovic <[email protected]> Co-authored-by: Scott Suarez <[email protected]>

google-cla bot added the cla: yes label Sep 8, 2021

kubasieron changed the title ~~Composer v2~~ Composer v2 (beta) (#9611) Sep 8, 2021

modular-magician requested a review from ScottSuarez September 8, 2021 15:08

kubasieron force-pushed the composer-v2 branch from 30bf583 to 1311063 Compare September 20, 2021 09:02

ScottSuarez requested changes Sep 21, 2021

View reviewed changes

kubasieron force-pushed the composer-v2 branch from efd026a to 9c362e1 Compare September 21, 2021 13:30

google-cla bot added cla: yes and removed cla: no labels Sep 28, 2021

ScottSuarez reviewed Sep 28, 2021

View reviewed changes

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb Show resolved Hide resolved

ScottSuarez reviewed Sep 28, 2021

View reviewed changes

composer v2 changes to enable cloud_composer_network_ipv4_cidr_block …

41113da

…to be set correctly

ScottSuarez and others added 5 commits October 4, 2021 16:40

safeguard null for expander

9ccb0f2

IPv4 Cidr working properly

58553fb

updated docs

642867a

typo fix

8a2fafe

Merge branch 'master' into composer-v2

e8d4ef7

specify beta for cloud_composer_network_ipv4_cidr_block

f3393ff

ScottSuarez approved these changes Oct 6, 2021

View reviewed changes

unless ga

0e63fa3

ScottSuarez merged commit 05e4e2c into GoogleCloudPlatform:master Oct 6, 2021

This was referenced Oct 6, 2021

Composer v2 (beta) (#9611) hashicorp/terraform-provider-google-beta#3709

Merged

Composer v2 (beta) (#9611) hashicorp/terraform-provider-google#10269

Merged

Composer v2 (beta) (#9611) #5181

Composer v2 (beta) (#9611) #5181

Conversation

kubasieron commented Sep 8, 2021 • edited by ScottSuarez Loading

modular-magician commented Sep 8, 2021

modular-magician commented Sep 8, 2021

Diff report:

ScottSuarez commented Sep 10, 2021

modular-magician commented Sep 10, 2021

Diff report:

ScottSuarez commented Sep 17, 2021

modular-magician commented Sep 17, 2021

Diff report:

modular-magician commented Sep 20, 2021

Diff report:

modular-magician commented Sep 20, 2021

modular-magician commented Sep 20, 2021

ScottSuarez commented Sep 20, 2021

modular-magician commented Sep 20, 2021

Diff report:

modular-magician commented Sep 20, 2021

modular-magician commented Sep 20, 2021

ScottSuarez commented Sep 21, 2021 • edited Loading

ScottSuarez left a comment

Choose a reason for hiding this comment

modular-magician commented Sep 21, 2021

Diff report:

modular-magician commented Sep 21, 2021

modular-magician commented Sep 21, 2021

modular-magician commented Sep 21, 2021

Diff report:

modular-magician commented Sep 21, 2021

modular-magician commented Sep 21, 2021

ScottSuarez commented Sep 21, 2021

ScottSuarez commented Sep 21, 2021

ScottSuarez commented Sep 21, 2021

ScottSuarez commented Sep 21, 2021

modular-magician commented Sep 28, 2021

Diff report:

ScottSuarez Sep 28, 2021

Choose a reason for hiding this comment

ScottSuarez Sep 28, 2021

Choose a reason for hiding this comment

ScottSuarez Sep 28, 2021

Choose a reason for hiding this comment

ScottSuarez commented Oct 4, 2021 • edited Loading

modular-magician commented Oct 4, 2021

Diff report:

modular-magician commented Oct 4, 2021

modular-magician commented Oct 4, 2021

ScottSuarez commented Oct 4, 2021

modular-magician commented Oct 5, 2021

Diff report:

modular-magician commented Oct 5, 2021

modular-magician commented Oct 5, 2021

Diff report:

modular-magician commented Oct 5, 2021

modular-magician commented Oct 6, 2021

Diff report:

modular-magician commented Oct 6, 2021

kubasieron commented Sep 8, 2021 •

edited by ScottSuarez

Loading

ScottSuarez commented Sep 21, 2021 •

edited

Loading

ScottSuarez commented Oct 4, 2021 •

edited

Loading