Switch to the new API for `marking as required` the fields for some resources #490

sergenyalcin · 2024-03-19T12:46:59Z

Description of your changes

This PR switches to the new API for marking as required the fields. The new API will mark as required the fields during the generation without any native resource schema change. For mor details of this new API please see this PR. The base strategies we decided in this PR:

Upjet's config.MarkAsRequired still modifies the native schema (no change in its implementation) but it's now deprecated in favor of config.Resource.MarkAsRequired.
Other clients of the config.MarkAsRequired are not updated to use the new config.Resource.MarkAsRequired in this PR because the config.MarkAsRequired modifies Terraform's runtime behavior (potentially also outside the context of the defaulting CustomizeDiff functions) by manipulating the native schema, and so we would like to test those resources which are updated to use the new config.Resource.MarkAsRequired. So it's a larger effort to transition all the clients of the config.MarkAsRequired and we limit the scope of this PR by only updating the configurations of the following resources.

Fixes #472

The root cause of #472 is native schema change of the zone and region field. The new CustomizeDiff functions of the underlying provider, DefaultProviderRegion and DefaultProviderZone, checks the Computed fields of the resource schemas and gives some decisions according to the values. At this point, because of manipulating the resource schema (what we must not do), we observe this issue. By this change, we will revert the schema changes for the affected resources. The affected resources were determined according to the two following rules:

The resource configuration should contain schema manipulation via MarkAsRequired for zone or region fields.
The native schema of resource should contain at least on of the mentioned CustomizeDiff functions.

Switched to the new API for the following resources:

Environment.composer
Job.cloudscheduler
Autoscaler.compute
Instance.compute
InstanceGroup.compute
InstanceGroupManager.compute
NetworkEndpointGroup.compute
ResourcePolicy.compute
TargetPool.compute
RegionInstanceGroupManager.compute
RegionAutoScaler.compute
ServiceAttachment.compute
EntryGroup.datacatalog

I have:

Read and followed Crossplane's contribution process.
Run make reviewable to ensure this PR is ready for review.
Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Instance.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8343310426
Job.cloudscheduler: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8343708323
InstanceGroup.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8343704491
ResourcePolicy.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8343851010
EntryGroup.datacatalog: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8343873861
RegionAutoScaler.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8344124389
NetworkEndpointGroup.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8344103249
RegionInstanceGroupManager.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8344548614
ServiceAttachment.compute: Manually (because of manual intervention need)
TargetPool.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8344649274
Autoscaler.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8345034404
InstanceGroupManager.compute: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8345034404
Environment.composer: Manually (because of service account permission issues)

Signed-off-by: Sergen Yalçın <[email protected]>

sergenyalcin · 2024-03-19T12:51:11Z

/test-examples="examples/compute/instance.yaml"

sergenyalcin · 2024-03-19T13:15:48Z

/test-examples="examples/compute/instancegroup.yaml"

sergenyalcin · 2024-03-19T13:16:02Z

/test-examples="examples/cloudscheduler/job.yaml"

sergenyalcin · 2024-03-19T13:24:12Z

/test-examples="examples/composer/environment.yaml"

sergenyalcin · 2024-03-19T13:24:50Z

/test-examples="examples/compute/autoscaler.yaml"

sergenyalcin · 2024-03-19T13:25:29Z

/test-examples="examples/compute/resourcepolicy.yaml"

sergenyalcin · 2024-03-19T13:27:12Z

/test-examples="examples/datacatalog/entrygroup.yaml"

sergenyalcin · 2024-03-19T13:43:03Z

/test-examples="examples/compute/networkendpointgroup.yaml"

sergenyalcin · 2024-03-19T13:44:35Z

/test-examples="examples/compute/regionautoscaler.yaml"

sergenyalcin · 2024-03-19T14:11:57Z

/test-examples="examples/compute/regioninstancegroupmanager.yaml"

sergenyalcin · 2024-03-19T14:18:04Z

/test-examples="examples/compute/targetpool.yaml"

…ix the example manifest of Environment.composer Signed-off-by: Sergen Yalçın <[email protected]>

sergenyalcin · 2024-03-19T14:39:49Z

/test-examples="examples/composer/environment.yaml"

sergenyalcin · 2024-03-19T14:41:50Z

/test-examples="examples/compute/autoscaler.yaml"

Signed-off-by: Sergen Yalçın <[email protected]>

ulucinar · 2024-03-19T16:00:44Z

config/compute/config.go

@@ -399,10 +399,6 @@ func Configure(p *config.Provider) { //nolint: gocyclo
 		config.MarkAsRequired(r.TerraformResource, "region")
 	})

-	p.AddResourceConfigurator("google_compute_region_per_instance_config", func(r *config.Resource) {


Does this resource no longer exist? Is this unnecessary?

I deleted this statement because this configuration does not have an effect. The region field is generated as a Reference field, and we mark the reference fields as Optional regardless of their status in the scheme. Also, as can be seen, even though the configuration was deleted, no diff occurred after generation. This shows that this configuration block has no function.

Thank you for the explanation.

ulucinar · 2024-03-19T16:01:16Z

examples/composer/environment.yaml

@@ -13,9 +13,3 @@ metadata:
 spec:
  forProvider:
    region: us-east1
-    config:


Why did we have to remove this configuration?

I checked the registry documentation, and this usage is no longer valid. Also, there was a basic usage in the example of registry. I changed this example to a basic one to avoid leading the user to the view that the config block is required. In this context, we can consider providing different example manifests for different purposes and use cases, which we discussed before.

When the relevant configuration is used in this form, error messages related to the cloud provider's own version change are seen. This points to a behavioral change in the cloud provider for the relevant resource. The relevant behavior change is not mentioned in the underlying provider documentation. In this context, we cannot rely on the relevant documentation to detect such behavioral changes in future underlying provider version updates. However, it may be possible to try to detect changes in the underlying provider's dependency on cloud providers.

ulucinar · 2024-03-19T16:49:05Z

Hi @sergenyalcin,
Let's also discuss in the PR description the strategy we decided on in crossplane/upjet#381:
a. Upjet's config.MarkAsRequired still modifies the native schema (no change in its implementation) but it's now deprecated in favor of config.Resource.MarkAsRequired.
b. This PR only updates & tests those resources whose configuration uses config.MarkAsRequired and their corresponding Terraform resources employ the new project and zone defaulting CustomizeDiffs -> This is already mentioned in the PR description.
c. Other clients of the config.MarkAsRequired are not updated to use the new config.Resource.MarkAsRequired in this PR because the config.MarkAsRequired modifies Terraform's runtime behavior (potentially also outside the context of the defaulting CustomizeDiff functions) by manipulating the native schema, and so we would like to test those resources which are updated to use the new config.Resource.MarkAsRequired. So it's a larger effort to transition all the clients of the config.MarkAsRequired and we limit the scope of this PR by only updating the configurations of the resources in [b].

ulucinar

Thanks @sergenyalcin, lgtm.

github-actions · 2024-03-20T12:07:28Z

Backport failed for release-1.0, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin release-1.0
git worktree add -d .worktree/backport-490-to-release-1.0 origin/release-1.0
cd .worktree/backport-490-to-release-1.0
git checkout -b backport-490-to-release-1.0
ancref=$(git merge-base be3db998f3d0479f02d24e2b10e3a6ef1ea45394 b9c3ab8a6b920162b31f7b039a734fc58e35dae2)
git cherry-pick -x $ancref..b9c3ab8a6b920162b31f7b039a734fc58e35dae2

Switch to the new API for marking as required the fields

60c0197

Signed-off-by: Sergen Yalçın <[email protected]>

sergenyalcin requested review from ulucinar and turkenf as code owners March 19, 2024 12:47

Fix external name configuration of InstanceGroupManager.compute and f…

3100942

…ix the example manifest of Environment.composer Signed-off-by: Sergen Yalçın <[email protected]>

sergenyalcin mentioned this pull request Mar 19, 2024

Add a new configuration option for required field generation crossplane/upjet#381

Merged

3 tasks

Bump upjet to v1.1.6

b9c3ab8

Signed-off-by: Sergen Yalçın <[email protected]>

sergenyalcin force-pushed the switch-to-new-required-api branch from 4097229 to b9c3ab8 Compare March 19, 2024 15:31

ulucinar reviewed Mar 19, 2024

View reviewed changes

ulucinar approved these changes Mar 20, 2024

View reviewed changes

sergenyalcin added the backport release-1.0 label Mar 20, 2024

sergenyalcin merged commit dec6492 into crossplane-contrib:main Mar 20, 2024
9 checks passed

sergenyalcin mentioned this pull request Mar 20, 2024

Bump upjet to 1.1.6 and switch to the new API for marking as required the fields for some resources #492

Merged

1 task

trampfox mentioned this pull request Jul 16, 2024

[Bug]: Cloud Function creation fails with SetNew only operates on computed keys - region is not one #575

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to the new API for `marking as required` the fields for some resources #490

Switch to the new API for `marking as required` the fields for some resources #490

sergenyalcin commented Mar 19, 2024 •

edited

Loading

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

ulucinar Mar 19, 2024

sergenyalcin Mar 20, 2024

ulucinar Mar 20, 2024

ulucinar Mar 19, 2024

sergenyalcin Mar 20, 2024 •

edited

Loading

ulucinar commented Mar 19, 2024

ulucinar left a comment

github-actions bot commented Mar 20, 2024

Switch to the new API for marking as required the fields for some resources #490

Switch to the new API for marking as required the fields for some resources #490

Conversation

sergenyalcin commented Mar 19, 2024 • edited Loading

Description of your changes

How has this code been tested

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

sergenyalcin commented Mar 19, 2024

ulucinar Mar 19, 2024

Choose a reason for hiding this comment

sergenyalcin Mar 20, 2024

Choose a reason for hiding this comment

ulucinar Mar 20, 2024

Choose a reason for hiding this comment

ulucinar Mar 19, 2024

Choose a reason for hiding this comment

sergenyalcin Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

ulucinar commented Mar 19, 2024

ulucinar left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 20, 2024

Switch to the new API for `marking as required` the fields for some resources #490

Switch to the new API for `marking as required` the fields for some resources #490

sergenyalcin commented Mar 19, 2024 •

edited

Loading

sergenyalcin Mar 20, 2024 •

edited

Loading